search

ESMValGroup / ESMValTool

ESMValTool: A community diagnostic and performance metrics tool for routine evaluation of Earth system models in CMIP
https://www.esmvaltool.org
Apache License 2.0
218 stars 127 forks source link

Regridding tripolar OSI-450-nh sea ice #2925

Open malininae opened 1 year ago

malininae commented 1 year ago

Hey folks,

as discussed during the November meeting I tried re-gridding the OSI-450-nh dataset onto regular 1x1 grid, but I get an error message with that. 

2022-11-17 21:13:48,786 UTC [1154413] ERROR   esmvalcore.preprocessor:338 Failed to run preprocessor function 'regrid' on the data
[<iris 'Cube' of sea_ice_area_fraction / (%) (time: 36; projection_y_coordinate: 432; projection_x_coordinate: 432)>]
loaded from original input file(s)
['/space/hall5/sitestore/eccc/crd/ccrn/users/emr001/OBS/Tier2/OSI-450-nh/OBS_OSI-450-nh_reanaly_v2_OImon_sic_197901-197912.nc',
 '/space/hall5/sitestore/eccc/crd/ccrn/users/emr001/OBS/Tier2/OSI-450-nh/OBS_OSI-450-nh_reanaly_v2_OImon_sic_198001-198012.nc',
 '/space/hall5/sitestore/eccc/crd/ccrn/users/emr001/OBS/Tier2/OSI-450-nh/OBS_OSI-450-nh_reanaly_v2_OImon_sic_198101-198112.nc',
 '/space/hall5/sitestore/eccc/crd/ccrn/users/emr001/OBS/Tier2/OSI-450-nh/OBS_OSI-450-nh_reanaly_v2_OImon_sic_198201-198212.nc']
(and 32 further file(s) not shown here; refer to the debug log for a full list)
with function argument(s)
target_grid = '1x1',
lon_offset = True,
lat_offset = True,
scheme = 'nearest'
2022-11-17 21:13:48,903 UTC [1154413] INFO    esmvalcore._task:127 Maximum memory used (estimate): 0.5 GB
2022-11-17 21:13:48,904 UTC [1154413] INFO    esmvalcore._task:129 Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur.
2022-11-17 21:13:48,904 UTC [1154413] ERROR   esmvalcore._main:511 Program terminated abnormally, see stack trace below for more information:
Traceback (most recent call last):
  File "/fs/homeu2/eccc/crd/ccrn_shr/emr001/ESMValCore/esmvalcore/_main.py", line 499, in run
    fire.Fire(ESMValTool())
  File "/space/hall5/sitestore/eccc/crd/ccrn/users/emr001/miniconda3/envs/esmvalcore_tool/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/space/hall5/sitestore/eccc/crd/ccrn/users/emr001/miniconda3/envs/esmvalcore_tool/lib/python3.10/site-packages/fire/core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/space/hall5/sitestore/eccc/crd/ccrn/users/emr001/miniconda3/envs/esmvalcore_tool/lib/python3.10/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/fs/homeu2/eccc/crd/ccrn_shr/emr001/ESMValCore/esmvalcore/_main.py", line 443, in run
    process_recipe(recipe_file=recipe, config_user=cfg)
  File "/fs/homeu2/eccc/crd/ccrn_shr/emr001/ESMValCore/esmvalcore/_main.py", line 124, in process_recipe
    recipe.run()
  File "/fs/homeu2/eccc/crd/ccrn_shr/emr001/ESMValCore/esmvalcore/_recipe.py", line 1921, in run
    self.tasks.run(max_parallel_tasks=self._cfg['max_parallel_tasks'])
  File "/fs/homeu2/eccc/crd/ccrn_shr/emr001/ESMValCore/esmvalcore/_task.py", line 723, in run
    self._run_sequential()
  File "/fs/homeu2/eccc/crd/ccrn_shr/emr001/ESMValCore/esmvalcore/_task.py", line 734, in _run_sequential
    task.run()
  File "/fs/homeu2/eccc/crd/ccrn_shr/emr001/ESMValCore/esmvalcore/_task.py", line 263, in run
    self.output_files = self._run(input_files)
  File "/fs/homeu2/eccc/crd/ccrn_shr/emr001/ESMValCore/esmvalcore/preprocessor/__init__.py", line 610, in _run
    product.apply(step, self.debug)
  File "/fs/homeu2/eccc/crd/ccrn_shr/emr001/ESMValCore/esmvalcore/preprocessor/__init__.py", line 421, in apply
    self.cubes = preprocess(self.cubes, step,
  File "/fs/homeu2/eccc/crd/ccrn_shr/emr001/ESMValCore/esmvalcore/preprocessor/__init__.py", line 357, in preprocess
    result.append(_run_preproc_function(function, item, settings,
  File "/fs/homeu2/eccc/crd/ccrn_shr/emr001/ESMValCore/esmvalcore/preprocessor/__init__.py", line 313, in _run_preproc_function
    return function(items, **kwargs)
  File "/fs/homeu2/eccc/crd/ccrn_shr/emr001/ESMValCore/esmvalcore/preprocessor/_regrid.py", line 634, in regrid
    cube = esmpy_regrid(cube, target_grid, scheme)
  File "/fs/homeu2/eccc/crd/ccrn_shr/emr001/ESMValCore/esmvalcore/preprocessor/_regrid_esmpy.py", line 341, in regrid
    regridder = build_regridder(src_rep, dst_rep, method)
  File "/fs/homeu2/eccc/crd/ccrn_shr/emr001/ESMValCore/esmvalcore/preprocessor/_regrid_esmpy.py", line 230, in build_regridder
    regridder = build_regridder_2d(src_rep, dst_rep,
  File "/fs/homeu2/eccc/crd/ccrn_shr/emr001/ESMValCore/esmvalcore/preprocessor/_regrid_esmpy.py", line 175, in build_regridder_2d
    src_field = cube_to_empty_field(src_rep)
  File "/fs/homeu2/eccc/crd/ccrn_shr/emr001/ESMValCore/esmvalcore/preprocessor/_regrid_esmpy.py", line 128, in cube_to_empty_field
    circular = is_lon_circular(lon)
  File "/fs/homeu2/eccc/crd/ccrn_shr/emr001/ESMValCore/esmvalcore/preprocessor/_regrid_esmpy.py", line 112, in is_lon_circular
    seam = (lon.bounds[1:-1, -1, (1, 2)]
TypeError: 'NoneType' object is not subscriptable

The OSI-450-nh is the file I downloaded from Levante, so it was a pre-cmorized for me version. I was wondering how to deal with that? Is it a known issue? Does the dataset need to be re-cmorized? Any help is highly appreciated. The recipe and the main_log_debug.txt are attached.

recipe_test_tripolar_regrid.yml.txt main_log_debug.txt

zklaus commented 1 year ago

In essence, this seems to show that the longitudes don't have bounds, which is a problem for non-regular grids. Can you confirm that longitudes (and likely then also latitudes) are boundless in the input file?

Side note: If you use triple backticks (``) to delineate code blocks, they will show up as real blocks with better formatting, compared with the simple backtick () that creates inline formatting and doesn't work well for multiline blocks.

malininae commented 1 year ago

@zklaus Indeed, I double checked and the OSI-450-Xh cubes seem to not have bounds around longitude an latitude coordinates. The issue is that cb.coord('longitude').guess_bounds() doesn't work, because the dimension is non-monotonic. What do you think would be the best way? Change CMORizer?

valeriupredoi commented 1 year ago

@zklaus Indeed, I double checked and the OSI-450-Xh cubes seem to not have bounds around longitude an latitude coordinates. The issue is that cb.coord('longitude').guess_bounds() doesn't work, because the dimension is non-monotonic. What do you think would be the best way? Change CMORizer?

yep, in principle - basically reproduce the logic and functionality behind guess_bounds() but for non-monotonic points - I think @schlunma did this once for some crazy data?

schlunma commented 1 year ago

guess_bounds only works for 1D data; I guess your lat/lon are 2D in this case.

We have a fix that works for 2D lat/lon here:

https://github.com/ESMValGroup/ESMValCore/blob/320a275d1730c229dce1e9921d03cb3ec1b18d97/esmvalcore/cmor/_fixes/common.py#L116

But since I am no expert on this at all I am not sure if this works in this case or is a sensible thing to do.

valeriupredoi commented 1 year ago

cheers, Manu! I reckon @zklaus is more of a 2D man than I am myself :grin:

malininae commented 1 year ago

@valeriupredoi, @schlunma cool, thanks! I never wrote fixes before, so would need some advice, what would be better, update the CMORizer or make/apply a fix?

schlunma commented 1 year ago

For observational datasets we usually update the CMORizer :+1: