search

CCI-Tools / cate

ESA CCI Toolbox (Cate)
MIT License
50 stars 15 forks source link

Coregistration of ocean colour data to SST dataset failing #733

Closed kjpearson closed 6 years ago

kjpearson commented 6 years ago

Expected behaviour

Ocean colour should be resampled onto the SST grid after resampling with coregister

Actual behaviour

"validate" option returns successfully "no problems encountered" but executing the coregister operation fails with:

Invalid Input

'Oops, Cate couldn't carry out the requested Executing operation "coregister" because:

Could not find a valid intersection to perform coregistration on'

There is then the traceback info but I can't find a way to copy that here. The last call is to coregistration.py, line340, in _find_intersection

Steps to reproduce the problem

For speed, I have created a subset of the SST CCI L4 v1.1 data for 2004-2005, lat=[-10,10], lon=[-175,-115] to act as the master dataset for coregister. This worked successfully with the cloud cci data as the slave dataset

I have then downloaded a 2004-01-01,2004-05-01 subset of the Ocean Colour CCI dataset "Global attenuation coefficient for downwelling irradiance (Kd490) gridded on a geographic projection V2

Attempting to coregister these two datasets immediately fails because there the ocean colour dataset has a variable "crs" with only a time dimension. I used select_var to extract "MERIS_nobs_sum" ad separately "Kd_940" but trying to coregister either of these caused the error above. I can't see what the problem is as both of these appear to be global datasets.

Specifications

MacOS, Cate 2.0.0-dev16

kjpearson commented 6 years ago

I get the same error when trying to coregister either the "active" or "passive" soil moisture datasets against a master global SST dataset. I also get it when trying to coregister the two soil moisture datasets against each other. I have taken 4 month temporal subsets in all three cases for 2004-01-01 to 2004-05-01. These are all globally gridded data sets.

kjpearson commented 6 years ago

I said above that the cloud cci variables successfully regrid but I'm having second thoughts. Although the coregistration operation completes, it looks like the data for the whole globe in the cloud dataset has been remapped down to the subregion I had in the SST dataset, rather than just those elements of the cloud dataset that were already within the boundary of the subregion. At least, that's how it looks to me from the patterns in the data. The "cee" and "cfc" variables seem the best to look at for this. Could you take a look at this too? If you agree, maybe it should be split to a different issue.

forman commented 6 years ago

Hi @JanisGailis could you please get back to @kjpearson

JanisGailis commented 6 years ago

@kjpearson I will investigate this. I have seen a similar effect before when doing spatial subset, but that turned out to be a visualization issue.

JanisGailis commented 6 years ago

@kjpearson The effect with global data being 'compressed' into the subset should be fixed in latest dev release. The problem with finding intersection still persists.

HelenClifton commented 5 years ago

Hi @kjpearson @forman @JanisGailis I'm retesting the coregistration with cate-2.0.0-dev.21 and still seeting several combinations of datasets where I'm getting the error capture Example traceback pasted in below Traceback (most recent call last): File "C:\Users\HClifton\cate-2.0.0.dev20\lib\site-packages\cate\util\web\jsonrpchandler.py", line 209, in send_service_method_result result = future.result() File "C:\Users\HClifton\cate-2.0.0.dev20\lib\concurrent\futures_base.py", line 425, in result return self.get_result() File "C:\Users\HClifton\cate-2.0.0.dev20\lib\concurrent\futures_base.py", line 384, in get_result raise self._exception File "C:\Users\HClifton\cate-2.0.0.dev20\lib\concurrent\futures\thread.py", line 56, in run result = self.fn(*self.args, *self.kwargs) File "C:\Users\HClifton\cate-2.0.0.dev20\lib\site-packages\cate\util\web\jsonrpchandler.py", line 306, in call_service_method result = method(method_params, monitor=monitor) File "C:\Users\HClifton\cate-2.0.0.dev20\lib\site-packages\cate\webapi\websocket.py", line 292, in set_workspace_resource monitor=monitor) File "C:\Users\HClifton\cate-2.0.0.dev20\lib\site-packages\cate\core\wsmanag.py", line 320, in set_workspace_resource workspace.execute_workflow(res_name=res_name, monitor=monitor) File "C:\Users\HClifton\cate-2.0.0.dev20\lib\site-packages\cate\core\workspace.py", line 662, in execute_workflow self.workflow.invoke_steps(steps, context=self._new_context(), monitor=monitor) File "C:\Users\HClifton\cate-2.0.0.dev20\lib\site-packages\cate\core\workflow.py", line 632, in invoke_steps step.invoke(context=context, monitor=monitor.child(work=1)) File "C:\Users\HClifton\cate-2.0.0.dev20\lib\site-packages\cate\core\workflow.py", line 318, in invoke self._invoke_impl(_new_context(context, step=self), monitor=monitor) File "C:\Users\HClifton\cate-2.0.0.dev20\lib\site-packages\cate\core\workflow.py", line 980, in _invoke_impl return_value = self._op(monitor=monitor, input_values) File "C:\Users\HClifton\cate-2.0.0.dev20\lib\site-packages\cate\core\op.py", line 216, in call return_value = self._wrapped_op(input_values) File "C:\Users\HClifton\cate-2.0.0.dev20\lib\site-packages\cate\ops\coregistration.py", line 140, in coregister return _resample_dataset(ds_master, ds_slave, methods_us[method_us], methods_ds[method_ds], monitor) File "C:\Users\HClifton\cate-2.0.0.dev20\lib\site-packages\cate\ops\coregistration.py", line 296, in _resample_dataset global_bounds=(-90, 90)) File "C:\Users\HClifton\cate-2.0.0.dev20\lib\site-packages\cate\ops\coregistration.py", line 341, in _find_intersection raise ValidationError('Could not find a valid intersection to perform' cate.core.types.ValidationError: Could not find a valid intersection to perform coregistration on

HelenClifton commented 5 years ago

@forman @JanisGailis Please could you re-open this issue and take a look. It's apparently an "Invalid Input" error so I don't know if it's related to the datasets or the code

JanisGailis commented 5 years ago

@HelenClifton Thanks for the report!

However, I won't reopen this issue, unless it's exactly the same datasets, as the error can be caused by many underlying data issues. Can you please open a new issue where you outline which datasets you're trying to coregister when you get the error?

JanisGailis commented 5 years ago

@HelenClifton One more thing. Norman did a release, but I'm not sure if the dev21 actually includes the latest changes from 'cate core', as he forgot to do something in that repository, as evidenced by C:\Users\HClifton\cate-2.0.0.dev20 in the stack trace. I don't know how the release process works in detail, but it is possible that the fix is not actually in the latest release yet.

@forman Correct me if I'm wrong.