Replacing Admin 1 Layers on IFRC GO Platform

[davidmuchatiza]

Description of the issue: The IFRC GO platform currently utilizes the Admin 0 and Admin 1 layers provided by ICRC for geographical data. However, it has been observed that some of the Admin 1 layers sourced from ICRC are outdated, with the last updates dating back to as early as 2007. Due to significant changes that have occurred in various countries since then, the accuracy and usefulness of these outdated Admin 1 layers have been compromised. This has particularly become evident when attempting to generate new DREF reports for countries where Admin 1 regions have changed such as splits or mergers between 2007 and the present.

Earlier this year, an effort was initiated to enhance the geographical data on the platform by incorporating Admin 2 layers from UN OCHA. However, this endeavor encountered challenges stemming from spatial discrepancies between the ICRC's Admin 0 and 1 layers and the OCHA dataset. Furthermore, since the OCHA dataset is more up-to-date in many countries, discrepancies arise where OCHA's Admin 1 regions differ in number from the outdated ICRC Admin 1 regions currently in use on the IFRC GO platform.

Notably, the Admin 1 data on the GO platform comprises various attributes, but a critical attribute requiring attention is the GO district ID, which serves as a unique identifier for each Admin 1 area within the GO platform's database. Recently, a successful test was conducted on the backend to assess the deprecation of certain Admin 1 regions. This test, overseen by Zoltan, proved promising in addressing the issue. However, a recent report from the Chilean Red Cross has highlighted a specific concern. The Nuble Region, which is an Admin 1 region added in September 2018, appears to be missing. This discrepancy underscores the urgency of our efforts to rectify the outdated Admin 1 data.

[davidmuchatiza]

We have agreed to address all Admin1 issues on a case-by-case basis. Presently, in the context of the Chile case, we intend to employ a two-step resolution process. We will deprecate the affected Admin1 area which is Biobio. Then introduce two new Admin1 areas: the newly defined BioBio and the Nuble region. The second step involves updating the geometries of the remaining Admin1 regions to ensure alignment with the newly inserted areas. During this update, we will assess whether any adjustments to the metadata are required. I've shared the Chile Admin1 shapefiles from both OCHA and our GO dataset. @geohacker will articulate the precise procedure for executing these tasks. It's worth noting that this process will serve as the standard approach for addressing similar cases in the future.

[nanometrenat]

xref prior (completed) Admin 1 issue re Pakistan https://github.com/IFRCGo/go-api/issues/1523

[geohacker]

@davidmuchatiza can you create a shapefile with the following columns for the new Chilean Admin1?

1. ADMIN01COD — this is what the codes were in ICRC shapefile. May be comparable to pcodes but you should confirm. For the admin1s that you want to update, the code should match exactly.
2. ADMIN01NAM — this should be what the names are
3. ISO2 — country iso
4. COUNTRY — country name

After that, @szabozoltan69 can help you deprecate the current Biobio admin 1.

The next step is importing the new admin1 and updating geometry for the existing ones. For creating the new Biobio geom:

1. Run python manage.py import-admin1-data <filename.shp>. This will create missing-districts.txt
2. Confirm that the Biobio is in the missing-districts.txt file.
3. Then, run python manage.py import-admin1-data <filename.shp> --import-missing missing-districts.txt. This will create the new Biobio admin1

Next we can update the geom, bbox and centroid for the existing admin1:

1. Run python manage.py import-admin1-data <filename.shp> --update-geom --update-bbox --update-centroid

@szabozoltan69 let's run this all on staging to confirm after we see the shapefile from @davidmuchatiza. Thank you!

[szabozoltan69]

On Staging that is Biobio: https://goadmin-stage.ifrc.org/en/admin/api/district/578 So I will make it deprecated when the moment arrives. (I am curious what will be the codename of the new area(s) instead of CL006.)

[davidmuchatiza]

@geohacker Thanks for the clear steps, but there's a minor challenge: the Admin1 Codes we are using aren't the same as the OCHA PCodes. If possible I suggest that we add another field on the current Admin1s on GO to house OCHA PCodes for future referencing. Then, for the current primary-foreign key link, I can add the Admin1 Codes to the OCHA data with the same values as they're on the ICRC dataset.

[szabozoltan69]

I had to include the not "is_deprecated" filter to import-admin1-data.py, instead of districts = District.objects.filter(code=code) let's use: districts = District.objects.filter(code=code).filter(is_deprecated=False)

With this change the following happens:

...manage.py import-admin1-data Chileadmin1update.shp
will write missing district codes to missing-districts.txt
done!

# cat missing-districts.txt 
code,name
CL006,Bío-Bío
CL006A,Ñuble

Then:

...manage.py import-admin1-data Chileadmin1update.shp --import-missing missing-districts.txt
will import these codes dict_keys(['CL006', 'CL006A'])
importing Bío-Bío
importing Ñuble
done!

Finally:

manage.py import-admin1-data Chileadmin1update.shp --update-geom --update-bbox --update-centroid
will write missing district codes to missing-districts.txt
done!

I wonder what happens with the 2 overlapping CL006 codes. :-D One is deprecated, one is not. Check it on https://goadmin-stage.ifrc.org/en/admin/api/district/?q=Chile Ñuble + Bío-Bío are the new, Biobio is the old one.

[szabozoltan69]

https://github.com/IFRCGo/go-api/commit/031c4fb234986b6724a7675e18f7b27bb501caef gives a fix for using non-deprecated areas.

[geohacker]

@szabozoltan69 ah that's a good catch to add the is_deprecated = False filter, thank you.

@davidmuchatiza I'd prefer to keep the code column and update it with the OCHA pcodes. This is our convention for admin2. Adding a new column would largely make things complicated for downstream map users. For the OCHA Chile pcode, can you see if there are any conflicts in the admin1 table on GO? If there are none, I'd prefer we update that in the process, of course that has to be done as a manual process after updating the geometries.

[nanometrenat]

xref previous discussions around OCHA P-codes vs IFRC codes: https://github.com/IFRCGo/go-frontend/issues/1838 Also https://github.com/IFRCGo/go-frontend/issues/1188#issuecomment-783406845