search

ESMValGroup / ESMValTool

ESMValTool: A community diagnostic and performance metrics tool for routine evaluation of Earth system models in CMIP
https://www.esmvaltool.org
Apache License 2.0
217 stars 126 forks source link

Suspect bug in NSIDC-0116-nh and NSIDC-0116-sh CMORizers: iris loading fails due to WKT projection not liked by pyproj #3203

Open TomasTorsvik opened 1 year ago

TomasTorsvik commented 1 year ago

Describe the bug I was trying to CMORize the NSIDC-0116-nh and NSIDC-0116-sh datasets using the standard esmvatool CMORization scripts. This fails with an error "pyproj.exceptions.CRSError":

For NSIDC-0116-nh: pyproj.exceptions.CRSError: Invalid projection: PROJCS["NSIDC EASE-Grid North",GEOGCS["Unspecified datum based upon the International 1924 Authalic Sphere",DATUM["Not_specified_based_on_International_1924_Authalic_Sphere",SPHEROID["International 1924 Authalic Sphere",6371228,0,AUTHORITY["EPSG","7057"]],AUTHORITY["EPSG","6053"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.01745329251994328,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4053"]],UNIT["metre",1,AUTHORITY["EPSG","9001"]],PROJECTION["Lambert_Azimuthal_Equal_Area"],PARAMETER["latitude_of_center",90],PARAMETER["longitude_of_center",0],PARAMETER["false_easting",0],PARAMETER["false_northing",0],AUTHORITY["EPSG","3408"],AXIS["X",UNKNOWN],AXIS["Y",UNKNOWN]]: (Internal Proj Error: proj_create: unhandled axis direction: UNKNOWN)

For NSIDC-0116-sh: pyproj.exceptions.CRSError: Invalid WKT string: PROJCS["NSIDC EASE-Grid South",GEOGCS["Unspecified datum based upon the International 1924 Authalic Sphere",DATUM["Not_specified_based_on_International_1924_Authalic_Sphere",SPHEROID["International 1924 Authalic Sphere",6371228,0,AUTHORITY["EPSG","7057"]],AUTHORITY["EPSG","6053"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.01745329251994328,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4053"]],UNIT["metre",1,AUTHORITY["EPSG","9001"]],PROJECTION["Lambert_Azimuthal_Equal_Area"],PARAMETER["latitude_of_center",-90],PARAMETER["longitude_of_center",0],PARAMETER["false_easting",0],PARAMETER["false_northing",0],AUTHORITY["EPSG","3409"],AXIS["X",UNKNOWN],AXIS["Y",UNKNOWN]]

From the error message, it seems that pyproj is not able to handle the CRS specified for the datasets. Looking into the EPSG database I find that the projections are marked as "superseded" EPSG-3408: https://epsg.org/crs_3408/NSIDC-EASE-Grid-North.html?sessionkey=1720ksyky2 EPSG-3409: https://epsg.org/crs_3409/NSIDC-EASE-Grid-South.html?sessionkey=1720ksyky2

I suppose this could explain why pyproj gives an error, but I'm not sure if it's the right explanation.

ESMValTool version:

Fails on:

Please attach main_log_debug.txt

TomasTorsvik commented 1 year ago

I tested with an older esmvaltool2.7.0 installation, and then the CMORizer worked. So it seems that something went wrong with the latest installation. Assuming this something is to do with proj/pyproj, the relevant parts may be

valeriupredoi commented 1 year ago

thanks a lot @TomasTorsvik for raising this! Unfortunately my knowledge of projections can be projected to a single point on a sheet of paper, but I went through proj and pyproj changelogs from the version that works to the one that doesn't and couldn't see anything labelled as deprecated/removed/breaking. The error comes from a compiled (pyx) module of pyproj - if it was indeed a Python issue that should have come from a Python script ie an exception raised by a chunk of Python code e.g. projection missing attributes/values etc, invalid type in proj definition etc. There are two aspects to this:

If @ESMValGroup/technical-lead-development-team have seen/found/have clues about this please chip in :beer:

TomasTorsvik commented 1 year ago

Hi @valeriupredoi , @ESMValGroup/technical-lead-development-team , the esmvaltool 2.8.0 installation that I used was instaled by the system admin, so I don't know how it was installed. I also tried to downgrade iris=3.4.1, but this is replaced by iris=3.6.0 when I do pip install --editable '.[develop]'.

However, I'm not sure if this is so relevant anymore. I did some further testing with the latest version of esmvaltool with ESMValCore 2.8.1 (esmvaltool-2.8.1 below), and with building from source from the main branch (esmvaltool-latest below). CMORization fails in both versions for NSIDC-0116-sh, and for NSIDC-0116-nh using esmvaltool-2.8.1, but works for NSIDC-0116-nh when I use esmvaltool-latest. :confused: The environments are built with mamba following the instructions in the documentation.

# esmvaltool-2.8.1
$ esmvaltool version
ESMValCore: 2.8.1
ESMValTool: 2.8.0

env_esmvaltool-2.8.1.txt NSIDC-0116-nh: http://ns2980k.web.sigma2.no/tomast/esmvaltool2.8.0/data_formatting_20230607_202555/run/main_log_debug.txt

# esmvaltool-latest
$ esmvaltool version
ESMValCore: 2.8.1
ESMValTool: 2.9.0.dev23+g54ace22f0

env_esmvaltool-latest.txt NSIDC-0116-nh: http://ns2980k.web.sigma2.no/tomast/esmvaltool2.8.0/data_formatting_20230607_203005/run/main_log_debug.txt NSIDC-0116-sh: http://ns2980k.web.sigma2.no/tomast/esmvaltool2.8.0/data_formatting_20230607_203533/run/main_log_debug.txt

valeriupredoi commented 1 year ago

cheers @TomasTorsvik - this debug is a lot better than the one before - sorry, not your fault or anything, it's just that this time round things don't come from a Cython compiled module and the call to pyproj is a lot more explicit, I see the issue with coordinate system, so I suspect this to be an iris=3.6.0 issue:

  File "/projects/NS2980K/home/tomast/conda/esmvaltool-latest/lib/python3.11/site-packages/iris/fileformats/_nc_load_rules/actions.py", line 172, in action_provides_grid_mapping
    coordinate_system = builder(engine, cf_var)
                        ^^^^^^^^^^^^^^^^^^^^^^^
  File "/projects/NS2980K/home/tomast/conda/esmvaltool-latest/lib/python3.11/site-packages/iris/fileformats/_nc_load_rules/helpers.py", line 719, in build_lambert_azimuthal_equal_area_coordinate_system
    ellipsoid = _get_ellipsoid(cf_grid_var)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/projects/NS2980K/home/tomast/conda/esmvaltool-latest/lib/python3.11/site-packages/iris/fileformats/_nc_load_rules/helpers.py", line 467, in _get_ellipsoid
    proj_crs = pyproj.crs.CRS.from_wkt(crs_wkt)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/projects/NS2980K/home/tomast/conda/esmvaltool-latest/lib/python3.11/site-packages/pyproj/crs/crs.py", line 431, in from_wkt
    raise CRSError(f"Invalid WKT string: {in_wkt_string}")

Now that you have a fully controllable environment built by yourself, would it be possible to simply mamba install -c conda-forge iris=3.4.1 in it and see if the cmorizer works with that older version? Re-installing iris with a different, older version should not change many dependencies. Cheers muchly :beer:

valeriupredoi commented 1 year ago

BTW it'd be rather useful if you got in touch with your sysadmins, when you got a bit of time, the standard paths to packages I see in your debug messages suggests to me that they do some weird stuff with the installation of different dependencies, one should almost never get an error from a .pyx script if that is compiled correctly too, maybe if you put me in touch with one of them we can talk how they do the installs :+1:

TomasTorsvik commented 1 year ago

Hi @valeriupredoi , we had some technical issues with the remote server so I wasn't able to run tests. I tried with mamba install -c conda-forge iris=3.4.1, but I get a conflict with something in the installation that requires iris>=3.6.0. 

$ esmvaltool data format NSIDC-0116-sh
ERROR:esmvalcore._main:Program terminated abnormally, see stack trace below for more information:
Traceback (most recent call last):
  File "/projects/NS2980K/home/tomast/conda/esmvaltool-latest/lib/python3.11/site-packages/esmvalcore/_main.py", line 521, in run
    fire.Fire(ESMValTool())
              ^^^^^^^^^^^^
  File "/projects/NS2980K/home/tomast/conda/esmvaltool-latest/lib/python3.11/site-packages/esmvalcore/_main.py", line 310, in __init__
    self.__setattr__(entry_point.name, entry_point.load()())
                                       ^^^^^^^^^^^^^^^^^^
  File "/projects/NS2980K/home/tomast/conda/esmvaltool-latest/lib/python3.11/site-packages/pkg_resources/__init__.py", line 2516, in load
    self.require(*args, **kwargs)
  File "/projects/NS2980K/home/tomast/conda/esmvaltool-latest/lib/python3.11/site-packages/pkg_resources/__init__.py", line 2539, in require
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/projects/NS2980K/home/tomast/conda/esmvaltool-latest/lib/python3.11/site-packages/pkg_resources/__init__.py", line 827, in resolve
    dist = self._resolve_dist(
           ^^^^^^^^^^^^^^^^^^^
  File "/projects/NS2980K/home/tomast/conda/esmvaltool-latest/lib/python3.11/site-packages/pkg_resources/__init__.py", line 873, in _resolve_dist
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (scitools-iris 3.4.1 (/projects/NS2980K/home/tomast/conda/esmvaltool-latest/lib/python3.11/site-packages), Requirement.parse('scitools-iris>=3.6.0'))

I can set iris=3.4.1 in the environment.yml file, but then I get back to the error that I had earlier.

valeriupredoi commented 1 year ago

yes the new esmvalcore=2.8.1 needs iris>3.6.0, you can install esmvalcore=2.8.0 that doesn't require such a new iris. But at any rate, I think I asked you to do way too many things, and would feel bad if I asked you to do more - any way I can grab hold of one of those raw OBS files myself so I can try it myself? :beer:

TomasTorsvik commented 1 year ago

@valeriupredoi Sure, go ahead :smiley:

valeriupredoi commented 1 year ago

OK I replicated the issue and it's not iris at fault :grin: - hear that @schlunma - you didn't break Iris :rofl:

valeriupredoi commented 1 year ago

I meant to say not iris changing version, keep an eye on this @TomasTorsvik https://github.com/SciTools/iris/issues/5343

rbeucher commented 1 year ago

I'm not even sure it is an Iris pb. Could be pyproj having issues with what is a specific polar projection. I was thinking of removing the axis information from the wkt. I haven't had the time though...

valeriupredoi commented 1 year ago

indeed so! Could well be, but I've not seen any issue at pyproj GH from others complaining about it - at the mo it's murky, I'd like to hear what iris folk have to say about it :beer:

valeriupredoi commented 1 year ago

just to keep peeps on track on this one, @trexfeathers has escalated this upstream with the PROJ folks https://github.com/pyproj4/pyproj/issues/1328

rbeucher commented 10 months ago

Hi All,

I tried today with the latest docker image and I still get the error. I thought this would be fixed with the new version of pyproj (3.6.1) but it isn't....

  File "/opt/conda/envs/esmvaltool/lib/python3.11/site-packages/pyproj/crs/crs.py", line 431, in from_wkt
    raise CRSError(f"Invalid WKT string: {in_wkt_string}")
pyproj.exceptions.CRSError: Invalid WKT string:  PROJCS["NSIDC EASE-Grid South",GEOGCS["Unspecified datum based upon the International 1924 Authalic Sphere",DATUM["Not_specified_based_on_International_1924_Authalic_Sphere",SPHEROID["International 1924 Authalic Sphere",6371228,0,AUTHORITY["EPSG","7057"]],AUTHORITY["EPSG","6053"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.01745329251994328,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4053"]],UNIT["metre",1,AUTHORITY["EPSG","9001"]],PROJECTION["Lambert_Azimuthal_Equal_Area"],PARAMETER["latitude_of_center",-90],PARAMETER["longitude_of_center",0],PARAMETER["false_easting",0],PARAMETER["false_northing",0],AUTHORITY["EPSG","3409"],AXIS["X",UNKNOWN],AXIS["Y",UNKNOWN]] 
2023-11-13 00:07:26,443 UTC [53] INFO    
If you have a question or need help, please start a new discussion on https://github.com/ESMValGroup/ESMValTool/discussions
If you suspect this is a bug, please open an issue on https://github.com/ESMValGroup/ESMValTool/issues
To make it easier to find out what the problem is, please consider attaching the files run/recipe_*.yml and run/main_log_debug.txt from the output directory.
rbeucher commented 10 months ago

Looks like we need to link to PROJ (9.3.0). Any issue with updating the conda environment?