CLIMADA-project / climada_python

Python (3.8+) version of CLIMADA
GNU General Public License v3.0
318 stars 125 forks source link

Natural Earth Update #284

Closed emanuel-schmid closed 3 years ago

emanuel-schmid commented 3 years ago

Natural Earth Data has updated their (?) API and data. We are using the service through cartopy.io.shapereader.natural_earth. Version 0.19.0.post1 of the cartopy library comes in two flavours, one that has the updated download url pattern, like in the master branch

    _NE_URL_TEMPLATE = ('https://naturalearth.s3.amazonaws.com/'
                        '{resolution}_{category}/ne_{resolution}_{name}.zip')

and one with the old pattern, which is not a valid url anymore:

    _NE_URL_TEMPLATE = ('https://naciscdn.org/naturalearth/{resolution}'
                        '/{category}/ne_{resolution}_{name}.zip')

After updating cartopy (conda update cartopy) the new template is in place and downloads are working again. But there is a side effect. Seemingly also the data changed and not just the url. This leads possibly to an inconsistent state, where the country names and id's don't match.

E.g., running the unit test climada.util.test.test_coordinates.TestGetGeodata.test_country_code_pass has produced this error:

ERROR: test_country_code_pass (climada.util.test.test_coordinates.TestGetGeodata)
Test set_region_id
----------------------------------------------------------------------
Traceback (most recent call last):
  File "...\climada_python\climada\util\test\test_coordinates.py", line 572, in test_country_code_pass
    region_id = u_coord.get_country_code(lat, lon, gridded=gridded)
  File "...\climada_python\climada\util\coordinates.py", line 1128, in get_country_code
    countries = get_country_geometries(extent=extent)
  File "...\climada_python\climada\util\coordinates.py", line 710, in get_country_geometries
    nat_earth.loc[idx, "ISO_N3"] = f"{natearth_country_to_int(country):03d}"
  File "...\climada_python\climada\util\coordinates.py", line 1091, in natearth_country_to_int
    return country_to_iso(str(country.NAME), representation="numeric")
  File "...\climada_python\climada\util\coordinates.py", line 1012, in country_to_iso
    raise LookupError(f'Unknown country identifier: {country}') from None
LookupError: Unknown country identifier: Brazilian I.

How this can be fixed needs to be clarified.

Assumedly one needs to delete the local cache of natural earth files after the update to the patched version. But before we know how to deal with the changed country identifiers it's probably advisable to stick to the old flavour - unless one doesn't have a local cache of natural earth files yet.

tovogt commented 3 years ago

See #296 and #313

tovogt commented 3 years ago

Fixed in #313

Note that the fix will work equally well for users that still have the old Natural Earth data in cache.