search

EC-Earth / ece2cmor3

Post-processing and cmorization of ec-earth output
Apache License 2.0
13 stars 6 forks source link

Broken IFS variables following yr frequency update #514

Closed oloapinivad closed 5 years ago

oloapinivad commented 5 years ago

Following the update of the yr frequencies (which causes the issue in NEMO cmorization discussed in #511) I see that 3 variables are missing in IFS cmorization.

I lost tasmin, tasmax (Amon) and sfcWindmax (Emon). I reinstalled from scratch ece2cmor3, run the git submodule update --init --recursive but still missing these 3 vars.

Browsing the logfile I found this:

2019-07-08 15:24:01 INFO:ece2cmor3.ifs2cmor: Post-processing variable 202 for target variable tasmin...
2019-07-08 15:24:01 ERROR:ece2cmor3.postproc: Unsupported combination of frequency mon with time operators [u'minimum within days', u'mean over days'] encountered

The 3 vars show the same issue.

goord commented 5 years ago

Hi Paolo, I don't fully understand what's going on there, it looks like on your system those time operators are read in as unicode strings. I checked in a commit that casts everything to python strings so can you check once more?

If everything is ok, you should definitely review the tasmin and tasmax cdo commands in the log file from previous successful cmorizations, and also run the test suite (type 'nosetests' in the ece2cmor3 checkout directory).

oloapinivad commented 5 years ago

Hi @goord just tested the last version but unfortunately does not work. NEMO is ok, but now IFS cmorisation crashes right after the filtering:

[...]
2019-07-09 06:51:08 INFO:ece2cmor3.grib_filter: Filtering grib file /scratch/ms/it/ccpd/tmp_cmor/chis_1930_9826/ifs_1930/data_1930/ICMSHchis+193012...
2019-07-09 06:51:43 ERROR:ece2cmor3.postproc: Unsupported combination of frequency 0 with time operators ['point'] encountered
Traceback (most recent call last):
  File "/perm/ms/it/ccpd/ecearth3/cmorization/ece2cmor3/ece2cmor3/ece2cmor.py", line 158, in <module>
    main()
  File "/perm/ms/it/ccpd/ecearth3/cmorization/ece2cmor3/ece2cmor3/ece2cmor.py", line 141, in main
    cdothreads=args.ncdo)
  File "/lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/ece2cmor3/lib/python2.7/site-packages/ece2cmor3-1.2.0-py2.7.egg/ece2cmor3/ece2cmorlib.py", line 186, in perform_ifs_tasks
    ifs2cmor.execute(ifs_tasks, nthreads=taskthreads)
  File "/lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/ece2cmor3/lib/python2.7/site-packages/ece2cmor3-1.2.0-py2.7.egg/ece2cmor3/ifs2cmor.py", line 187, in execute
    read_mask(task.target.variable, getattr(task, cmor_task.output_path_key))
AttributeError: 'cmor_task' object has no attribute 'path'

I have checked older logfiles from previous years and everything is ok. Indeed I successfully ran QA-DKRZ and nctime on the first 80 years of data (historical AOGCM). Since yer 81 I am testing the new code and I am getting the mentioned issues.

goord commented 5 years ago

Sorry Paolo, this should be fixed in 6f22a750f178a35fc7db703bb013928d9f93bcf6

oloapinivad commented 5 years ago

Thanks Gijs, not it is fixed! Closing!