search

EC-Earth / ece2cmor3

Post-processing and cmorization of ec-earth output
Apache License 2.0
13 stars 6 forks source link

Release 0.9.9: attributeError: 'cmor_task' object has no attribute 'path' #449

Closed oloapinivad closed 5 years ago

oloapinivad commented 5 years ago

Hi all,

i am testing the new prelease with a fresh checkout from the master. NEMO is working fine but I am having issue with IFS using data from a test historical simulation. Surprisingly this is not happening for the first year of the simulation, but only for the second one. The files seem all correctly linked so I am quite puzzled. The error is coming from April, which is somehow unexpected: it seems CDO is trying to load a file which has not been filtered. Here is the log:

2019-04-15 10:33:46 INFO:ece2cmor3.grib_filter: Filtering grib file /scratch/ms/it/ccpd/tmp_cmor/cj02_1851_14687/ifs_1851/linkdata/ICMGGcj02+185012...
2019-04-15 10:33:46 INFO:ece2cmor3.grib_filter: Filtering grib file /scratch/ms/it/ccpd/tmp_cmor/cj02_1851_14687/ifs_1851/linkdata/ICMSHcj02+185012...
2019-04-15 10:34:27 INFO:ece2cmor3.grib_filter: Filtering grib file /scratch/ms/it/ccpd/tmp_cmor/cj02_1851_14687/ifs_1851/linkdata/ICMGGcj02+185101...
2019-04-15 10:35:01 INFO:ece2cmor3.grib_filter: Filtering grib file /scratch/ms/it/ccpd/tmp_cmor/cj02_1851_14687/ifs_1851/linkdata/ICMSHcj02+185101...
2019-04-15 10:35:07 INFO:ece2cmor3.grib_filter: Filtering grib file /scratch/ms/it/ccpd/tmp_cmor/cj02_1851_14687/ifs_1851/linkdata/ICMGGcj02+185102...
2019-04-15 10:35:55 INFO:ece2cmor3.grib_filter: Filtering grib file /scratch/ms/it/ccpd/tmp_cmor/cj02_1851_14687/ifs_1851/linkdata/ICMGGcj02+185103...
2019-04-15 10:36:27 INFO:ece2cmor3.grib_filter: Filtering grib file /scratch/ms/it/ccpd/tmp_cmor/cj02_1851_14687/ifs_1851/linkdata/ICMSHcj02+185102...
2019-04-15 10:36:49 INFO:ece2cmor3.grib_filter: Filtering grib file /scratch/ms/it/ccpd/tmp_cmor/cj02_1851_14687/ifs_1851/linkdata/ICMGGcj02+185104...
2019-04-15 10:37:40 INFO:ece2cmor3.grib_filter: Filtering grib file /scratch/ms/it/ccpd/tmp_cmor/cj02_1851_14687/ifs_1851/linkdata/ICMGGcj02+185105...
2019-04-15 10:37:40 INFO:ece2cmor3.grib_filter: Filtering grib file /scratch/ms/it/ccpd/tmp_cmor/cj02_1851_14687/ifs_1851/linkdata/ICMSHcj02+185103...
2019-04-15 10:38:25 INFO:ece2cmor3.grib_filter: Filtering grib file /scratch/ms/it/ccpd/tmp_cmor/cj02_1851_14687/ifs_1851/linkdata/ICMGGcj02+185106...
2019-04-15 10:38:57 INFO:ece2cmor3.grib_filter: Filtering grib file /scratch/ms/it/ccpd/tmp_cmor/cj02_1851_14687/ifs_1851/linkdata/ICMSHcj02+185104...
2019-04-15 10:39:05 INFO:ece2cmor3.grib_filter: Filtering grib file /scratch/ms/it/ccpd/tmp_cmor/cj02_1851_14687/ifs_1851/linkdata/ICMGGcj02+185107...
2019-04-15 10:39:47 INFO:ece2cmor3.grib_filter: Filtering grib file /scratch/ms/it/ccpd/tmp_cmor/cj02_1851_14687/ifs_1851/linkdata/ICMGGcj02+185108...
2019-04-15 10:40:17 INFO:ece2cmor3.grib_filter: Filtering grib file /scratch/ms/it/ccpd/tmp_cmor/cj02_1851_14687/ifs_1851/linkdata/ICMSHcj02+185105...
2019-04-15 10:40:35 INFO:ece2cmor3.grib_filter: Filtering grib file /scratch/ms/it/ccpd/tmp_cmor/cj02_1851_14687/ifs_1851/linkdata/ICMGGcj02+185109...
2019-04-15 10:41:22 INFO:ece2cmor3.grib_filter: Filtering grib file /scratch/ms/it/ccpd/tmp_cmor/cj02_1851_14687/ifs_1851/linkdata/ICMGGcj02+185110...
2019-04-15 10:41:36 INFO:ece2cmor3.grib_filter: Filtering grib file /scratch/ms/it/ccpd/tmp_cmor/cj02_1851_14687/ifs_1851/linkdata/ICMSHcj02+185106...
2019-04-15 10:42:01 INFO:ece2cmor3.grib_filter: Filtering grib file /scratch/ms/it/ccpd/tmp_cmor/cj02_1851_14687/ifs_1851/linkdata/ICMGGcj02+185111...
2019-04-15 10:42:49 INFO:ece2cmor3.grib_filter: Filtering grib file /scratch/ms/it/ccpd/tmp_cmor/cj02_1851_14687/ifs_1851/linkdata/ICMGGcj02+185112...
2019-04-15 10:42:50 INFO:ece2cmor3.grib_filter: Filtering grib file /scratch/ms/it/ccpd/tmp_cmor/cj02_1851_14687/ifs_1851/linkdata/ICMSHcj02+185107...
2019-04-15 10:43:50 INFO:ece2cmor3.grib_filter: Filtering grib file /scratch/ms/it/ccpd/tmp_cmor/cj02_1851_14687/ifs_1851/linkdata/ICMSHcj02+185108...
2019-04-15 10:44:29 INFO:ece2cmor3.grib_filter: Filtering grib file /scratch/ms/it/ccpd/tmp_cmor/cj02_1851_14687/ifs_1851/linkdata/ICMSHcj02+185109...
2019-04-15 10:45:07 INFO:ece2cmor3.grib_filter: Filtering grib file /scratch/ms/it/ccpd/tmp_cmor/cj02_1851_14687/ifs_1851/linkdata/ICMSHcj02+185110...
2019-04-15 10:45:42 INFO:ece2cmor3.grib_filter: Filtering grib file /scratch/ms/it/ccpd/tmp_cmor/cj02_1851_14687/ifs_1851/linkdata/ICMSHcj02+185111...
2019-04-15 10:46:15 INFO:ece2cmor3.grib_filter: Filtering grib file /scratch/ms/it/ccpd/tmp_cmor/cj02_1851_14687/ifs_1851/linkdata/ICMSHcj02+185112...
2019-04-15 10:46:50 INFO:ece2cmor3.postproc: Post-processing target land in table fx from file /scratch/ms/it/ccpd/tmp_cmor/cj02_1851_14687/ifs_1851/linkdata/ICMGGcj02+185104 with cdo command -setgridtype,regular -seltimestep,1 -selcode,172
2019-04-15 10:46:51 ERROR:ece2cmor3.cdoapi: (returncode:1) cdo(2) seltimestep: Process started
cdo(3) selcode: Process started
Warning (cgribexScanTimestep2): Record 57 (id=49.128 lev1=0 lev2=0) timestep 2: Parameter not defined at timestep 1!
cdo(3) selcode: Open failed on >/scratch/ms/it/ccpd/tmp_cmor/cj02_1851_14687/ifs_1851/linkdata/ICMGGcj02+185104<
Unsupported file structure

Traceback (most recent call last):
  File "/perm/ms/it/ccpd/ecearth3/cmorization/ece2cmor3/ece2cmor3/ece2cmor.py", line 153, in <module>
    main()
  File "/perm/ms/it/ccpd/ecearth3/cmorization/ece2cmor3/ece2cmor3/ece2cmor.py", line 136, in main
    cdothreads=args.ncdo)
  File "/lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/ece2cmor3/lib/python2.7/site-packages/ece2cmor3-0.9.9-py2.7.egg/ece2cmor3/ece2cmorlib.py", line 185, in perform_ifs_tasks
    ifs2cmor.execute(ifs_tasks, nthreads=taskthreads)
  File "/lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/ece2cmor3/lib/python2.7/site-packages/ece2cmor3-0.9.9-py2.7.egg/ece2cmor3/ifs2cmor.py", line 181, in execute
    read_mask(task.target.variable, getattr(task, cmor_task.output_path_key))
AttributeError: 'cmor_task' object has no attribute 'path'
goord commented 5 years ago

Hi Paolo, can you try again after copying (or linking) the initial state file ICMGGcj02+000000 to the directory of the second year? I believe ece2cmor3 is trying to find that file for the land mask, but somehow doesn't find it (probably because you have a symlink in your path) and then tries to extract the land mask from the file ICMGGcj02+185104. However, because of the staggered time step structure of this file, cdo can't read it...

oloapinivad commented 5 years ago

Great, thanks. Linking the initial state file now works.

However I have a problem with the linked files: I am cmorizing year 1851 from a historical run starting in 1850. If I link the December of the previous year (185012) in the same folder the resulting file starts from the 1st of December instead of the 1st January. See the naming of the file below (record within corresponds):

zg500_6hrPlevPt_EC-Earth3_piControl_r1i1p1f1_gr_185012010000-185112311800.nc

How can I tell ece2cmor to read the 1851-01-01 00:00:00? If I do not link the file I get the warning of a missing point: 

2019-04-16 07:08:58 WARNING:ece2cmor3.ifs2cmor: The file /scratch/ms/it/ccpd/tmp_cmor/cj02_1851_13387/ifs_1851/CMIP6/cj02-ifs-1850/zg500_6hrPlevPt.nc seems to be missing 1 time stamps at the beginning, these will be added

Thanks, Paolo

goord commented 5 years ago

Hi Paolo,

ece2cmor3 searches for the previous year by going 1 directory up and then searching for the previous leg, following the output structure of the EC-Earth run directory. So if you want to maintain that, I suggest you create a link to the entire output directory instead of individual legs, and pointing the directory argument to the leg in the linked folder.

oloapinivad commented 5 years ago

Hi Gijs,

at CNR we do not use the same output structure, which means that we do not split output by leg but only by year (Year_1850, Year_1851, etc). Leg sequence does not exist in our output structure. Do you think that I should build the same directory tree with the leg numbering (000, 001, etc) to make ece2cmor3 work? Is there any other way to hack (for example providing to ifs2cmor.initialise the path of the previous leg)?

goord commented 5 years ago

Hi Paolo,

I think this should not be a problem, as I recall ece2cmor3 just goes 1 directory up and then searches recursively for the output of the previous month. So as long as your link includes all yearly subdirectories, this should work.

If it is not working, could you then post the log output file and a schematic layout of your output structure (including links)?

oloapinivad commented 5 years ago

Great, after linking the full structure it works! Closing this.