warnings!

[abby-baskind]

@jbusecke I did what you recommended at the end of our meeting today and it worked about as well as we expected it to (yay!). of course there were warnings aplenty. here's the full code but i'll try to summarize the warnings that came up

1. So for this block of code

variables = ['dissic','talk', 'so', 'thetao']
z_kwargs = {'consolidated': True, 'use_cftime': True}
query = dict(experiment_id=['historical'], 
#              table_id=['Omon'], 
             variable_id=variables,
             grid_label=['gr', 'gn'],
              source_id=['GFDL-CM4', 'GFDL-ESM4', 'GFDL-OM4p5B', 'IPSL-CM6A-LR',
                         'CNRM-ESM2-1', 'CanESM5', 'CESM2', 'CESM2-WACCM', 'CanESM5-CanOE',
                         'UKESM1-0-LL', 'MPI-ESM-1-2-HAM', 'MPI-ESM1-2-LR', 'MPI-ESM1-2-HR',
                         'GISS-E2-1-G', 'NorESM2-LM', 'GISS-E2-1-G-CC', 'MIROC-ES2L',
                         'NorCPM1', 'NorESM1-F', 'NorESM2-MM', 'ACCESS-ESM1-5', 'CESM2-FV2',
                         'CESM2-WACCM-FV2', 'MRI-ESM2-0', 'IPSL-CM5A2-INCA', 'KIOST-ESM',
                         'EC-Earth3-CC', 'CMCC-ESM2', 'IPSL-CM6A-LR-INCA']
            )

cat = col.search(**query)

# print(cat.df['source_id'].unique())
dset_dict_old = cat.to_dataset_dict(zarr_kwargs=z_kwargs, storage_options={'token': 'anon'},
                                preprocess=combined_preprocessing, aggregate=False)

The warnings looked like this

 /srv/conda/envs/notebook/lib/python3.8/site-packages/cmip6_preprocessing/preprocessing.py:211: UserWarning: MIROC-ES2L: No units found for lev
  warnings.warn(f'{ds.attrs["source_id"]}: No units found for {co}')
/srv/conda/envs/notebook/lib/python3.8/site-packages/cmip6_preprocessing/preprocessing.py:211: UserWarning: MIROC-ES2L: No units found for lev
  warnings.warn(f'{ds.attrs["source_id"]}: No units found for {co}')
/srv/conda/envs/notebook/lib/python3.8/site-packages/cmip6_preprocessing/preprocessing.py:286: UserWarning: Found time as dimension in `lon_bounds`. Assuming this is an error and just picking the first step along that dimension.
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.8/site-packages/cmip6_preprocessing/preprocessing.py:286: UserWarning: Found time as dimension in `lev_bounds`. Assuming this is an error and just picking the first step along that dimension.
  warnings.warn(

2. For merge_variables It looked like all (or at least most) of the warnings had to do with time so here are some of the warnings

/srv/conda/envs/notebook/lib/python3.8/site-packages/cmip6_preprocessing/postprocessing.py:122: UserWarning: CMIP.NCC.NorCPM1.historical.r18i1p1f1.Omon.gr.none failed to combine with :indexes along dimension 'time' are not equal
  warnings.warn(f"{cmip6_dataset_id(ds)} failed to combine with :{e}")
/srv/conda/envs/notebook/lib/python3.8/site-packages/cmip6_preprocessing/postprocessing.py:122: UserWarning: CMIP.NCC.NorCPM1.historical.r1i1p1f1.Omon.gr.none failed to combine with :indexes along dimension 'time' are not equal
  warnings.warn(f"{cmip6_dataset_id(ds)} failed to combine with :{e}")
/srv/conda/envs/notebook/lib/python3.8/site-packages/cmip6_preprocessing/postprocessing.py:122: UserWarning: CMIP.CNRM-CERFACS.CNRM-ESM2-1.historical.r2i1p1f2.Omon.gn.none failed to combine with :cannot compare cftime.DatetimeProlepticGregorian(1850, 1, 16, 12, 0, 0, 0) and cftime.DatetimeGregorian(1850, 1, 16, 12, 0, 0, 0) (different calendars)

But the good news is that at least for some models variables were in fact merged and data sets indeed were combined! Some of them are gn so they might have to be regridded but that's an issue for another day. Overall, I'm taking this as a win 🥳

[jbusecke]

Awesome! The warnings in 1) can totally be ignored (they will disspear in a version in the future).

In 2) we are dealing with two issues: Some of the models probably have an issue with duplicate times, and another one uses two different calendars (🙄).

for the latter I suggest this simple fix: Augment the preprocessing function like this:

from cmip6_preprocessing.drift_removal import replace_time
def new_preprocessing(ds):
    ds = replace_time(ds) #This will replace the time with a standard calendar. For monthly data the calendars do not matter much
    ds = combined_preprocessing(ds) #This is what would happen if you pass `preprocess=combined_preprocessing` to `.to_dset_dict`
    return ds

...

...to_dset_dict(..., preprocess=new_preprocessing)

...

Could you try this and let me know if these (cannot compare cftime.DatetimeProlepticGregorian(1850, 1, 16, 12, 0, 0, 0) and cftime.DatetimeGregorian(1850, 1, 16, 12, 0, 0, 0) (different calendars)) types of warnings disspear?

I am still trying to figure out how to elegantly solve 1)...might take a bit more time.

[abby-baskind]

So the calendar warning still comes up (I think twice and for the same models as the previous attempt but here's an example)

/srv/conda/envs/notebook/lib/python3.8/site-packages/cmip6_preprocessing/postprocessing.py:122: UserWarning: CMIP.IPSL.IPSL-CM6A-LR.historical.r10i1p1f1.Omon.gn.none failed to combine with :cannot compare cftime.DatetimeProlepticGregorian(1850, 1, 1, 0, 0, 0, 0) and cftime.DatetimeGregorian(1850, 1, 1, 0, 0, 0, 0) (different calendars)

the published code has been updated/committed and i implemented your suggestion at the very bottom if you want to look that over for more info

[gmacgilchrist]

Is it just the calendar warning that comes up? i.e. the other warning failed to combine with :indexes along dimension 'time' are not equal has been solved by this fix?

It looks to me like the calendar warning is only appearing for one member of the CNRM model, with other members of the same model passing through no problem? Since we're selecting just one member anyway, I would suggest that unless there is an obvious hack, you can just leave this particular member out. It looks like you're getting a lot more models than before anyway, which is great.