search

CCI-Tools / cate

ESA CCI Toolbox (Cate)
MIT License
50 stars 15 forks source link

Time coordinates not correctly decoded for some Aerosol products #771

Open forman opened 6 years ago

forman commented 6 years ago

Expected behavior

Aerosol CCI data's time coordinate variable should be of type datetime64[ns] after decoding attributes.

Actual behavior

Aerosol CCI data's time dimension is (still) of type int32, so no decoding took place.

This may be due to lacking of related CF-metadata in the data products.

Steps to reproduce the problem

  1. Open Aerosol CCI "esacci.AEROSOL.mon.L3.AAI.multi-sensor.multi-platform.ms_uvai.1-5-7.r1"
  2. Inspect time dimension

Specifications

Cate 2.0.0-dev.19

forman commented 6 years ago

May be related to #623, #634, and most importantly #664.

forman commented 6 years ago

In fact, it seems there is no information in Aerosol CCI data that would allow Cate to decode time information.

Must inform Aerosol CCI team about that.

forman commented 6 years ago

@hans-permana could you please investigate a bit deeper, e.g. checking all available Aerosol data types?

hans-permana commented 6 years ago

It seems like there are inconsistencies on the existence of time_coverage_start and time_coverage_end attributes which are required to construct the right time dimension:

Dataset Issue Cause
ESA Aerosol Climate Change Initiative (Aerosol CCI): Level 2 aerosol products from MERIS (ALAMO algorithm), Version 2.2 Error "variable latitude not equal across datasets" (no issue # yet) See stack trace below. Note, dataset has only time_coverage_start set.
ESA Aerosol Climate Change Initiative (Aerosol CCI): Level 3 aerosol products from the Multi-Sensor UVAI algorithm (MS UVAI), Version 1.5.7 (climatology) Error as reported in https://github.com/CCI-Tools/cate/issues/664#issuecomment-395367240
ESA Aerosol Climate Change Initiative (Aerosol CCI): Level 3 aerosol products from the Multi-Sensor UVAI algorithm (MS UVAI), Version 1.5.7(daily) no time decoding no time_coverage_start nor time_coverage_end. There is a date information on the description.
ESA Aerosol Climate Change Initiative (Aerosol CCI): Level 3 aerosol products from the Multi-Sensor UVAI algorithm (MS UVAI), Version 1.5.7(monthly) no time decoding no time_coverage_start nor time_coverage_end. There is a date information on the description.
ESA Aerosol Climate Change Initiative (Aerosol CCI): Level 3 aerosol products from AATSR (ORAC algorithm), Version 3.02 (monthly) -
ESA Aerosol Climate Change Initiative (Aerosol CCI): Level 3 aerosol products from AATSR (ORAC algorithm), Version 3.02 (daily) -
ESA Aerosol Climate Change Initiative (Aerosol CCI): Level 3 aerosol products from AATSR (SU algorithm), Version 4.21 (monthly) -
ESA Aerosol Climate Change Initiative (Aerosol CCI): Level 3 aerosol products from AATSR (SU algorithm), Version 4.21 (daily) -
ESA Aerosol Climate Change Initiative (Aerosol CCI): Level 3 aerosol products from ATSR2 (ORAC algorithm), Version 3.02 (monthly) -
ESA Aerosol Climate Change Initiative (Aerosol CCI): Level 3 aerosol products from ATSR2 (ORAC algorithm), Version 3.02 (daily) -
ESA Aerosol Climate Change Initiative (Aerosol CCI): Level 3 aerosol products from ATSR2 (SU algorithm), Version 4.21 (monthly) -
ESA Aerosol Climate Change Initiative (Aerosol CCI): Level 3 aerosol products from ATSR2 (SU algorithm), Version 4.21 (daily) -
ESA Aerosol Climate Change Initiative (Aerosol CCI): Level 3 aerosol products from GOMOS (AERGOM algorithm), Version 2.19 (days) no time decoding no time_coverage_start nor time_coverage_end. There are startDate and endDate in float.
ESA Aerosol Climate Change Initiative (Aerosol CCI): Level 3 aerosol products from MERIS (ALAMO algorithm), Version 2.2 (monthly) -
ESA Aerosol Climate Change Initiative (Aerosol CCI): Level 3 aerosol products from MERIS (ALAMO algorithm), Version 2.2 (daily) -
forman commented 6 years ago

Stack trace for ESA Aerosol Climate Change Initiative (Aerosol CCI): Level 2 aerosol products from MERIS (ALAMO algorithm), Version 2.2 (esacci.AEROSOL.satellite-orbit-frequency.L2P.AOD.MERIS.Envisat.MERIS_ENVISAT.2-2.r1):

Traceback (most recent call last):
  File "D:\Projects\cate\cate\ds\local.py", line 183, in open_dataset
    monitor=monitor)
  File "D:\Projects\cate\cate\core\ds.py", line 667, in open_xarray_dataset
    **kwargs)
  File "D:\Miniconda3\envs\cate-env\lib\site-packages\xarray\backends\api.py", line 642, in open_mfdataset
    data_vars=data_vars, coords=coords)
  File "D:\Miniconda3\envs\cate-env\lib\site-packages\xarray\core\combine.py", line 436, in auto_combine
    for ds in grouped]
  File "D:\Miniconda3\envs\cate-env\lib\site-packages\xarray\core\combine.py", line 436, in <listcomp>
    for ds in grouped]
  File "D:\Miniconda3\envs\cate-env\lib\site-packages\xarray\core\combine.py", line 365, in _auto_concat
    return concat(datasets, dim=dim, data_vars=data_vars, coords=coords)
  File "D:\Miniconda3\envs\cate-env\lib\site-packages\xarray\core\combine.py", line 120, in concat
    return f(objs, dim, data_vars, coords, compat, positions)
  File "D:\Miniconda3\envs\cate-env\lib\site-packages\xarray\core\combine.py", line 276, in _dataset_concat
    'variable %s not equal across datasets' % k)
ValueError: variable latitude not equal across datasets

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\Projects\cate\cate\util\web\jsonrpchandler.py", line 209, in send_service_method_result
    result = future.result()
  File "D:\Miniconda3\envs\cate-env\lib\concurrent\futures\_base.py", line 425, in result
    return self.__get_result()
  File "D:\Miniconda3\envs\cate-env\lib\concurrent\futures\_base.py", line 384, in __get_result
    raise self._exception
  File "D:\Miniconda3\envs\cate-env\lib\concurrent\futures\thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "D:\Projects\cate\cate\util\web\jsonrpchandler.py", line 306, in call_service_method
    result = method(*method_params, monitor=monitor)
  File "D:\Projects\cate\cate\webapi\websocket.py", line 292, in set_workspace_resource
    monitor=monitor)
  File "D:\Projects\cate\cate\core\wsmanag.py", line 320, in set_workspace_resource
    workspace.execute_workflow(res_name=res_name, monitor=monitor)
  File "D:\Projects\cate\cate\core\workspace.py", line 662, in execute_workflow
    self.workflow.invoke_steps(steps, context=self._new_context(), monitor=monitor)
  File "D:\Projects\cate\cate\core\workflow.py", line 627, in invoke_steps
    steps[0].invoke(context=context, monitor=monitor)
  File "D:\Projects\cate\cate\core\workflow.py", line 318, in invoke
    self._invoke_impl(_new_context(context, step=self), monitor=monitor)
  File "D:\Projects\cate\cate\core\workflow.py", line 980, in _invoke_impl
    return_value = self._op(monitor=monitor, **input_values)
  File "D:\Projects\cate\cate\core\op.py", line 216, in __call__
    return_value = self._wrapped_op(**input_values)
  File "D:\Projects\cate\cate\ops\io.py", line 83, in open_dataset
    monitor=monitor)
  File "D:\Projects\cate\cate\core\ds.py", line 589, in open_dataset
    return data_source.open_dataset(time_range, region, var_names, monitor=monitor.child(20))
  File "D:\Projects\cate\cate\ds\local.py", line 197, in open_dataset
    raise ValidationError(msg) from e
cate.core.types.ValidationError: Cannot open local dataset for time range 2008-01-01, 2008-01-02T23:59:59:
variable latitude not equal across datasets