Unpin intake-esm, new version activated validation of optional fields and broke our notebooks

[tlvu]

Found in https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/pull/106

Related issue https://github.com/intake/intake-esm/issues/549

Error found by Jenkins:

20:31:53  _________ pavics-sdi-master/docs/source/notebooks/climex.ipynb::Cell 0 _________
20:31:53  Notebook cell execution failed
20:31:53  Cell 0: Cell execution caused an exception
20:31:53  
20:31:53  Input:
20:31:53  import xarray as xr
20:31:53  import intake
20:31:53  import xclim
20:31:53  from xclim import ensembles as xens
20:31:53  from clisops.core.subset import subset_gridpoint
20:31:53  from dask.diagnostics import ProgressBar
20:31:53  from dask.distributed import Client, LocalCluster
20:31:53  from IPython.display import Markdown, HTML
20:31:53  import shutil
20:31:53  
20:31:53  
20:31:53  cat = intake.open_esm_datastore("https://pavics.ouranos.ca/catalog/climex.json")  # TEST_USE_PROD_DATA
20:31:53  cat.df.head()
20:31:53  
20:31:53  Traceback:
20:31:53  
20:31:53  ---------------------------------------------------------------------------
20:31:53  ValidationError                           Traceback (most recent call last)
20:31:53  Cell In [1], line 12
20:31:53        8 from IPython.display import Markdown, HTML
20:31:53        9 import shutil
20:31:53  ---> 12 cat = intake.open_esm_datastore("https://pavics.ouranos.ca/catalog/climex.json")  # TEST_USE_PROD_DATA
20:31:53       13 cat.df.head()
20:31:53  
20:31:53  File /opt/conda/envs/birdy/lib/python3.8/site-packages/intake_esm/core.py:94, in esm_datastore.__init__(self, obj, progressbar, sep, registry, read_csv_kwargs, storage_options, intake_kwargs)
20:31:53       92     self.esmcat = ESMCatalogModel.from_dict(obj)
20:31:53       93 else:
20:31:53  ---> 94     self.esmcat = ESMCatalogModel.load(
20:31:53       95         obj, storage_options=self.storage_options, read_csv_kwargs=read_csv_kwargs
20:31:53       96     )
20:31:53       98 self.derivedcat = registry or default_registry
20:31:53       99 self._entries = {}
20:31:53  
20:31:53  File /opt/conda/envs/birdy/lib/python3.8/site-packages/intake_esm/cat.py:255, in ESMCatalogModel.load(cls, json_file, storage_options, read_csv_kwargs)
20:31:53      253 if 'last_updated' not in data:
20:31:53      254     data['last_updated'] = None
20:31:53  --> 255 cat = cls.parse_obj(data)
20:31:53      256 if cat.catalog_file:
20:31:53      257     if _mapper.fs.exists(cat.catalog_file):
20:31:53  
20:31:53  File /opt/conda/envs/birdy/lib/python3.8/site-packages/pydantic/main.py:526, in pydantic.main.BaseModel.parse_obj()
20:31:53  
20:31:53  File /opt/conda/envs/birdy/lib/python3.8/site-packages/pydantic/main.py:342, in pydantic.main.BaseModel.__init__()
20:31:53  
20:31:53  ValidationError: 1 validation error for ESMCatalogModel
20:31:53  aggregation_control
20:31:53    field required (type=value_error.missing)

[huard]

Can this be closed ?

[tlvu]

Can this be closed ?

Well, only if we can lift the pin https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/0a918bdc95319ea1394cba6b979994295473e3dc/docker/environment.yml#L109-L113

Meaning we have to test if the new version works properly.

[aulemahal]

In theory the two latest versions of intake-esm have the fix.
So

- intake-esm >= 2023.6.14

should work.

[tlvu]

I think this is fixed by https://github.com/intake/intake-esm/pull/569 so this issue will be closed by the next Jupyter env rebuild, see commit 33c1ec7e35bc68d610ff08714b5e0dcd9459101a.