Map Export v1

[batpad]

This is a bit of a higher level ticket for the v1 of the "Map Export" tool - we can break these out into smaller tickets if required.

For this iteration, we want to add functionality for the user to see a map on the Emergency Detail page, highlighting the affected country and admin1 boundaries within it. They should also be able to "Export" this map - i.e. be able to export to an image for print, etc.

For this we will need:

• Ability to associate Emergencies with Districts in the database (Many2Many relation)
• Ability for user to associate Emergencies with Districts in the form to add Field Reports.
• Highlight the associated districts on the emergency pages, on a custom designed Mapbox map.
• Repurpose the "Print Map" functionality that @mankamolnar has been working on to export the map and surrounding HTML / design elements.

We also probably want the ability to have logos of the National Society involved on the map. I think this would involve adding a logo field or so to the Country model: https://github.com/IFRCGo/go-api/blob/master/api/models.py#L71 - for now, we can probably add the logos in the Django Admin, so we should not need any frontend to enable National Societies to add logos.

cc @szabozoltan69 @ElsaRaunioIFRC

[batpad]

I realize I misunderstood the database structure a little bit when writing the above ticket.

The current structure has Emergencies (this is Events in the database), and within it, it can have many Field Reports.

I imagine we want the map to be exportable for the Emergency, hence it might not make sense to have the step to choose the Districts as part of the Field Report form. Is there a separate step where Emergencies are created? Ideally, the districts would be specified to be associated with the Emergencies? And then we can show them on the map on the Emergency page as per the designs and make it exportable.

Let me know if this makes sense ^ and if this is something @szabozoltan69 either you or @ElsaRaunioIFRC can help with - i.e. figuring out where we store the association to Districts and also where will we ask users to add this data about which Districts this Emergency pertains to.

[szabozoltan69]

Well, in the process of the Field Report creation there is a decision: do we link it to an existing Emergency (=Event) or not (in this case the backend will create a new emergency with the Field Report's name).

You are correct, the districts should be associated with Emergencies.

[batpad]

@szabozoltan69 interesting.

So perhaps we need to give the user the option to add districts in the Field Report form, and have some logic like:

• If Emergency (=Event) does not exist, create it and associate districts that user has added.
• If Emergency exists, add any new districts user has selected in the Field Report form to it.

Is there a possibility ever for the same Emergency to have associations with Districts belonging to multiple Countries? If yes, @ElsaRaunioIFRC we would need to think of what this means for the map design while exporting. Also would make the form a little bit more complicated (since we can't assume that all the districts will be chosen from a single country).

@szabozoltan69 it would be helpful for me to understand the relationship between Emergency and FieldReport a little bit better and how the different fields there get populated - perhaps we can go over on the call or you can send me any notes you think might be useful for me to understand.

[szabozoltan69]

@batpad Thanks for your lines. This subject (FieldReport - > Emergency) has a history until now, but there is also a desire to change the workflow a bit (mainly: to keep more fields in Emergency, and how to update them in case of new/changed FieldReports). I'll give you more feedback on this soon.

[szabozoltan69]

On API side, when the create_field_report is called, this code part is used: https://github.com/IFRCGo/go-api/blob/master/api/drf_views.py#L495-L511

There was a plan so that a field report update should update related event also, like "num_injured", " num_missing"..., which exist both in field_report and in event, but recently they are not updated. I suppose we should put them into the https://github.com/IFRCGo/go-api/blob/master/api/drf_views.py#L556-L595 part, but (of course) frontend should also be changed to get these data (instead of the Field Reports) from Events.

[batpad]

Updating this ticket with some notes from @ElsaRaunioIFRC -

Link to Mapbox style to use: https://api.mapbox.com/styles/v1/go-ifrc/cjxa3k4cx39a21cqt9qilk9hp.html?fresh=true&title=true&access_token=pk.eyJ1IjoiZ28taWZyYyIsImEiOiJjamlybHUzNmMwZDUxM3Fwa215YjJ1dzk5In0.OhqEybDX303sHjHzq3nFYQ#6.39/42.906/22.935

Some additional notes / complexities / things to keep in mind:

One of the challenges in the map generation process is to ensure the proper display of disputed territories. Disputed boundaries should be clearly distinguished from the defined international boundaries (as you can see on the map). During the coding process, there are some additional considerations, including:

• When a user selects Serbia, Kosovo (and Admin1 regions under Kosovo) should also be included in the selection. In the code, I guess it would look like this: Kosovo > Serbia

• When a user selects Morocco, Western Sahara should also be included, like this: Western Sahara > Morocco

• Territories marked as Disputed or Unclaimed (under the “Country” column in the ICRC Admin0 dataset) should be marked with diagonal line filling. Since I did not find such an option in the MapBox default styles, I have used triangles instead (I know it looks silly on the map but it is just for demonstration purposes).

• We also need to discuss how to deal with countries that have a lot of overseas territories (e.g. France and UK). If there’s a disaster in New Caledonia (French island in the Pacific), the user must select France as the country and New Caledonia as Admin1. In this case, the map should be automatically adjusted to the appropriate location. (I can provide a sketch map on this)

[batpad]

We may need to add some relationships in the database to define these country relations, eg. cases like Kosovo > Serbia.

To solve the New Caledonia type problem might be a bit tricky - currently, we don't store any geospatial data about either countries or admin1 boundaries in the GO-Api database - these exist on Mapbox and we pull the geo-data in on the front-end maps where required. But to deal with these cases, we may need to store a centroid for Admin1 boundaries, so that we are able to position the map more accurately (since we cannot rely on zooming to country borders because of edge cases like New Caledonia).

[szabozoltan69]

We could define a "belongs_to" table, which can have only these special values (entity, country).

[ElsaRaunioIFRC]

@batpad @szabozoltan69: I agree that a "belongs_to" table/column would be essential for managing the relationships between Admin0s and Admin1s. Should I proceed to compile such a column? We already have the data under the "Country" column on the Admin0 database; with MATCH and INDEX, we could transform the data to the Admin1 dataset too.

[batpad]

@ElsaRaunioIFRC We already have the mapping from admin0 to admin1 entities - in our database we call it Country and District and for each Country, we know which Districts are a part of it. (Of course, we use the word "District", but it could be any admin1 boundary).

I think here we need to think about how to represent cases like "When a user selects Morocco, Western Sahara should also be included, like this: Western Sahara > Morocco". For this, something like how @szabozoltan69 described, mapping "entity" to "country", where entity could be something like Western Sahara sounds like what we need.

[ElsaRaunioIFRC]

@batpad Could you share me the link to the country database on GitHub? I would like to check that we are consistent in defining a "country". I would like to check that none of the non-independent entities is classified as a country.

Yesterday, someone from the Netherlands Red Cross contacted us asking why we are listing "Aruba" as a country. Aruba is not listed on the front page but you can still find it with the search function. I would like to clean the database in order to solve issues like this.

[batpad]

@szabozoltan69 is it possible to export the api_country and api_district tables from the production database for @ElsaRaunioIFRC to look through? I believe one can also view and edit these entries in the Django admin.

We should probably start a separate ticket on cleaning up the existing data in the database.

Also, to note: Currently we do not store any geospatial information for countries or districts in the database. Just the codes, and then we can look it up on the frontend on the Mapbox map. However, it it possible that we may want to start storing some geospatial information - either polygons or centroids for countries / districts in the database. @ElsaRaunioIFRC maybe when you look at the existing data, we can have a conversation about this.

[ElsaRaunioIFRC]

Hi Sanjay,

Referring to this conversation on GitHub, I have taken a new look at our current spatial data on Admin1 areas, available here: https://drive.google.com/open?id=1lUQIooJQXF3xVlxtdRuPiZBpfVBe0VGh

I’ve updated the “Admin00Nam” and “Country” columns to ensure that each Admin1 area is associated with the correct sovereign country. In cases where the sovereignty is not clear, the area has been marked as “Disputed” or “Unclaimed”.

I’m currently also in the process of uploading this shapefile to the mapbox.

Additionally, I have attached some sketches about dealing with the overseas territories. As you can see, I propose indicating the affected territory and the sovereign country in the map labels, for example Aruba (Netherlands).

Let me know if you need any additional data/other help from my side.

Best regards,

Elsa

From: Sanjay Bhangar notifications@github.com Sent: 27 June 2019 09:27 To: IFRCGo/go-api go-api@noreply.github.com Cc: Elsa Katariina RAUNIO Elsa.RAUNIO@ifrc.org; Mention mention@noreply.github.com Subject: Re: [IFRCGo/go-api] Map Export v1 (#437)

@szabozoltan69https://github.com/szabozoltan69 is it possible to export the api_country and api_district tables from the production database for @ElsaRaunioIFRChttps://github.com/ElsaRaunioIFRC to look through? I believe one can also view and edit these entries in the Django admin.

We should probably start a separate ticket on cleaning up the existing data in the database.

Also, to note: Currently we do not store any geospatial information for countries or districts in the database. Just the codes, and then we can look it up on the frontend on the Mapbox map. However, it it possible that we may want to start storing some geospatial information - either polygons or centroids for countries / districts in the database. @ElsaRaunioIFRChttps://github.com/ElsaRaunioIFRC maybe when you look at the existing data, we can have a conversation about this.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/IFRCGo/go-api/issues/437?email_source=notifications&email_token=AHNG6365RGUXFL54YKDQA3TP4RTTDA5CNFSM4H23ITB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYWGTJA#issuecomment-506227108, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AHNG633EXBT5W5TLDAHB4I3P4RTTDANCNFSM4H23ITBQ.

[szabozoltan69]

I'll make a dump from countries and districts soon...

[szabozoltan69]

countries_20190627.xlsx districts_20190627.xlsx Here are they.

[ElsaRaunioIFRC]

@szabozoltan69 Thank you for sharing these files. We will definitely need a "belongs_to" column (or similar) in the data as we can see entities like Puerto Rico and Greenland classified as countries. If there's anything I can do with this, let me know.

[szabozoltan69]

In https://github.com/IFRCGo/go-api/blob/master/api/management/commands/ingest_appeals.py#L39-L41 there are a few "belongs_to" information, but it is not complete. For example Puerto Rico and Greenland is not there... If you have ideas how to make this list for "irregular territories", let's start it.

[ElsaRaunioIFRC]

I could complement this list by pulling the data from our excel file and adding it into the code. Shall we proceed as this?

[szabozoltan69]

Good Idea!

[batpad]

As a first step, updating the data in this list is probably a good idea to get all the data in one place.

@szabozoltan69 it would be nice to think about if we should represent this in the database. It will be nice to have this data as part of the database - we could then include it in queries, etc.

Maybe we could get all the data together and all the different cases in front of us, and we could get on a call on Monday to go over what different scenarios exist, and then come up with a proper data model to represent these?

[szabozoltan69]

Correct – a database representation would be much appreciated. Also the scenarios should be cleaned up, yes. :-)

[batpad]

Had a very useful discussion today.

To summarize what we will do here:

• Add a type field for Country in the database to identify whether it is "Not a Country" (eg IFRC), "A Country" (the normal case) - or "A region" eg. "Southern Asia Country Cluster". We will add the field, make it editable in the Django Admin, and @ElsaRaunioIFRC will help us entering this data.

• Add an is_enclave field for District in the database. This will identify District entities like New Caledonia which are far away from the parent country. Then we will have logic on the frontend so that if an Emergency applies to a District that is an Enclave, we zoom to the boundaries of the District rather than the parent Country.

• Add a districts column to FieldReports, to be able to associate Field Reports with Districts. We can then fetch all the Districts associated with all FieldReports for a particular Emergency to show on the map.

• For special cases where we need to select multiple countries when the user selects one country, we decided that this is only 2 cases so far, and we can just handle them as special cases in the frontend.

In addition, we will check data integrity between data in the database and data on Mapbox, and make recommendations to add other standard ISO codes to the data.

cc @geohacker

[nanometrenat]

xref https://github.com/IFRCGo/go-frontend/issues/938