Several questions to long_term_average

[HerzogStephan]

Info: Currently I try to calculate the LTA as a reference for the enso_nino34-operation.

Approaches: CLI: cate --traceback res set cl_lta long_term_average source='esacci.CLOUD.mon.L3C.CLD_PRODUCTS.AVHRR.NOAA-17.AVHRR_NOAA.1-0.r1' year_min=2007 year_max=2008 file='/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/UC06/cl_lta.nc' var='cc_total'

Returns:
LTA: done

....traceback_start
Traceback (most recent call last): File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/util/cli.py", line 274, in run_main args_obj.command_class().execute(args_obj) File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/util/cli.py", line 171, in execute return sub_command_function(command_args) File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/cli/main.py", line 945, in _execute_set monitor=cls.new_monitor()) File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/webapi/wsmanag.py", line 171, in set_workspace_resource data=self._post_data(op_name=op_name, op_args=json.dumps(op_args))) File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/webapi/wsmanag.py", line 63, in _fetch_json raise error_type(message) cate.core.workspace.WorkspaceError: ('Unsupported protocol', None)

Cate WebAPI service traceback

Traceback (most recent call last): File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/webapi/rest.py", line 240, in post monitor=_new_monitor()) File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/core/wsmanag.py", line 307, in set_workspace_resource workspace.execute_workflow(res_name=res_name, monitor=monitor) File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/core/workspace.py", line 520, in execute_workflow Workflow.invoke_steps(steps, value_cache=self._resource_cache, monitor=monitor) File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/core/workflow.py", line 577, in invoke_steps steps[0].invoke(value_cache=value_cache, monitor=monitor) File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/core/workflow.py", line 911, in invoke return_value = self._op_registration(monitor=monitor, input_values) File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/core/op.py", line 225, in call return_value = operation(input_values) File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/ops/average.py", line 124, in long_term_average data_source.sync(dt_range, monitor=monitor.child(work=step * 0.9)) File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/ds/esa_cci_odp.py", line 481, in sync raise ValueError('Unsupported protocol', protocol) ValueError: ('Unsupported protocol', None)

cate res: error: ('Unsupported protocol', None)

Cate WebAPI service traceback

Traceback (most recent call last): File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/webapi/rest.py", line 240, in post monitor=_new_monitor()) File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/core/wsmanag.py", line 307, in set_workspace_resource workspace.execute_workflow(res_name=res_name, monitor=monitor) File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/core/workspace.py", line 520, in execute_workflow Workflow.invoke_steps(steps, value_cache=self._resource_cache, monitor=monitor) File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/core/workflow.py", line 577, in invoke_steps steps[0].invoke(value_cache=value_cache, monitor=monitor) File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/core/workflow.py", line 911, in invoke return_value = self._op_registration(monitor=monitor, input_values) File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/core/op.py", line 225, in call return_value = operation(input_values) File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/ops/average.py", line 124, in long_term_average data_source.sync(dt_range, monitor=monitor.child(work=step * 0.9)) File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/ds/esa_cci_odp.py", line 481, in sync raise ValueError('Unsupported protocol', protocol) ValueError: ('Unsupported protocol', None) ....traceback-end

API: from cate.core.ds import DATA_STORE_REGISTRY import cate.ops as ops from cate.util import ConsoleMonitor

monitor = ConsoleMonitor() data_store = DATA_STORE_REGISTRY.get_data_store('esa_cci_odp') local_store = DATA_STORE_REGISTRY.get_data_store('local')

name_oz = 'esacci.OZONE.mon.L3.NP.multi-sensor.multi-platform.MERGED.fv0002.r1' tr_uc6 = '2007-01-01, 2008-12-31' vr_names_oz = 'O3_du_tot' ozone_remote_sources = data_store.query(name_oz) ozone_remote_sources[0].make_local(local_name=name_oz, time_range=tr_uc6, var_names=vr_names_oz, monitor=monitor) #year 2006

name_cl = 'esacci.CLOUD.mon.L3C.CLD_PRODUCTS.AVHRR.NOAA-17.AVHRR_NOAA.1-0.r1' vr_names_cl = 'cc_total' cloud_remote_sources = data_store.query(name_cl) cloud_remote_sources[0].make_local(local_name=name_cl, time_range=tr_uc6, var_names=vr_names_cl, monitor=monitor)

oz06 = ops.open_dataset('local.' + name_oz) cl06 = ops.open_dataset('local.' + name_cl)

Co-Register:

cores_output = ops.coregister(oz06,cl06) cl06 = cores_output

Subset:

region_uc6 = '-170,-5,-120,5' oz_tmp = ops.subset_spatial(ds=oz06,region=region_uc6) #,mask=bool) cl_tmp = ops.subset_spatial(ds=cl06,region=region_uc6) #,mask=bool)

cl_lta = ops.long_term_average(name_cl,var=vr_names_cl, year_min=2007, year_max=2008, file='/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/UC06/cl_lta.nc', save=True, monitor=monitor) LTA: started LTA: 0% LTA: done

....traceback-start Traceback (most recent call last): File "", line 1, in File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/ops/average.py", line 124, in long_term_average data_source.sync(dt_range, monitor=monitor.child(work=step * 0.9)) File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/ds/esa_cci_odp.py", line 481, in sync raise ValueError('Unsupported protocol', protocol) ValueError: ('Unsupported protocol', None) ....traceback-end

Question: Can someone explain me what I did wrong?

[JanisGailis]

I know what's going on. Cate 8 rc1 is too old. It still has the old long_term_average operation that tries to fetch data on it's own, and it's a mismatch with the new ds implementation.

@mzuehlke Can we make an rc2 release with the current state of the software?

Another option would be if the installed cate-core can somehow be updated from master. Is this possible? This would be great, as then Stephan and co would always have the latest state of the software, yet wouldn't have to bother with manual installation!

[HerzogStephan]

In addition it seems that other operations are also not updated: E.g. temporal_agg: CLI: cate op info temporal_agg Returns: […]Operation cate.ops.average.temporal_agg […]

Isn't it the case that .average.py is obsolete and it belongs to .aggregate.py by now?

[JanisGailis]

OK, so I think you should be able to update cate from git. I think you have already checked out the repository, so git checkout master, git pull. As you have installed cate, you should have a conda installation with all the required packages installed. So, you should just be able to do python setup.py install in cate-core root directory, where setup.py is located.

Then you should have an up to date cate-core.

[HerzogStephan]

the installer does not automatically make it a git-rep.

[JanisGailis]

Yes. So checkout the repository (cate-core) somewhere else, unrelated to the installation. And do python setup.py install there. I think (hehe :)) it should overwrite the installed cate-core in your conda environment.

The crucial thing here is that setup.py is being executed with the correct python installation. In general though, if the command line cate runs, then running the above line should over-write the installed version.

[HerzogStephan]

On github->cate-core/cate/ops are all the .py’s listed which later on go into the ‘/cate-core/lib/python3.5/site-packages/cate/ops/’-directory – right?

The directory setup by the installer does not contain the same list of .py’s as listed on github. When I pull the latest version of the software into a directory named 'checkout_master' I do find a very different list of .py's (operations) within the directory 'checkout_master/cate-core/cate/ops' (Even though I did 'python setup.py install', which updated some parts of ./bin from the installer.)

That is why I think I did something wrong or it did not work. Maybe I need a new installer ...

[JanisGailis]

Yes, I think that's the correct directory. And I would've thought it should've updated that directory when you ran 'python setup.py install'.

Ah, so maybe you can try to give the full path to the correct python installation? I don't have it installed through an installer on Linux, but on windows it's

C:/User/name/cate/python.exe setup.py install

While located in the directory checked out from master.

[HerzogStephan]

I use the python that comes with the installer:

/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/bin/python

[HerzogStephan]

I do not know what went wrong trying updating it the first time. However, now I do have an updated TBX containing the latest OP-versions. Thanks for that!