search

EC-Earth / ece2cmor3

Post-processing and cmorization of ec-earth output
Apache License 2.0
13 stars 6 forks source link

IndexError thrown for some IFS variables #428

Closed treerink closed 5 years ago

treerink commented 5 years ago

As offline reported by @ufladrich the cmorisation of mrsol gives an IndexError: list index out of range error which seems equal to the one earlier reported in #291.

That the error comes up now is clear, we recently added the preferences (the prefs.py) and mrsol has been added inin the prefs.py file even more recently in e0e8dc576098f8a066a36c2088798e00894fcafe in order to give preference to IFS for this variable.

goord commented 5 years ago

Hi Thomas can Uwe post his log file?

ufladrich commented 5 years ago

Here is a summary:

I am using cmip6-data-request-varlist-CMIP-historical-EC-EARTH-Veg.json as varlist.json file. When I use the original file from r6705-control-output-files (at r6720) and run something like

ece2cmor \
  --ifs \
  --conf ../cmorise.metadata.ifs.json \
  --ececonf EC-EARTH-Veg \
  --vars [...]/cmip6-data-request-varlist-CMIP-historical-EC-EARTH-Veg.json \
  --exp b601 \
  --npp 32 \
  --tmpdir [...] \
  --log \
  --odir [...] \
  --skip_alevel_vars \
  [...]/ifs/010

I get:

Traceback (most recent call last):
  File "[...]/envs/ece2cmor3/bin/ece2cmor", line 11, in <module>
    load_entry_point('ece2cmor3==0.9.0', 'console_scripts', 'ece2cmor')()
  File "[...]/envs/ece2cmor3/lib/python2.7/site-packages/ece2cmor3-0.9.0-py2.7.egg/ece2cmor3/ece2cmor.py", line 136, in main
    cdothreads=args.ncdo)
  File "[...]/envs/ece2cmor3/lib/python2.7/site-packages/ece2cmor3-0.9.0-py2.7.egg/ece2cmor3/ece2cmorlib.py", line 185, in perform_ifs_tasks
    ifs2cmor.execute(ifs_tasks, nthreads=taskthreads)
  File "[...]/envs/ece2cmor3/lib/python2.7/site-packages/ece2cmor3-0.9.0-py2.7.egg/ece2cmor3/ifs2cmor.py", line 189, in execute
    pool.map(cmor_worker, tasks)
  File "[...]/envs/ece2cmor3/lib/python2.7/multiprocessing/pool.py", line 253, in map
    return self.map_async(func, iterable, chunksize).get()
  File "[...]/envs/ece2cmor3/lib/python2.7/multiprocessing/pool.py", line 572, in get
    raise self._value
IndexError: list index out of range

When I remove ifs-Emon-mrsol from the varlist.json file, my job completes fine. When I remove everything but mrsol, I still get the above error.

I attach the job and ece2cmor logs for the two cases. No. 2332507/...162400 is just-mrsol and 2331938/...143943 is everything-but-mrsol.

ece2cmor-job-2331938-10.log ece2cmor-job-2332507-10.log b601-ifs-010-20190327143943.log b601-ifs-010-20190327162400.log

goord commented 5 years ago

In the log I see the error

2019-03-27 17:33:54 ERROR:ece2cmor3.postproc: Merging expressions of multiple variables per layer is not supported.

Will figure out what's going on...

goord commented 5 years ago

This recently broke due to one of my commits. Should be fixed in rev. ad9a708e33510bf4b170aee1db9b284c4e7a2ff6

ufladrich commented 5 years ago

I still seem to have the same problem. Including mrsol gives the IndexError. However, I think that the Merging expressions of multiple variables... error is gone.

I will re-do a test with only mrsol and post the logs. Just have to wait for my current test to finish...

ufladrich commented 5 years ago

Unfortunately, it appears to be more difficult than that: I get the error despite removing mrsol. This time, while cmorising a scenarioMIP-ssp126 experiment. To clarify: I run this test without mrsol, but maybe that is not relevant.

ufladrich commented 5 years ago

... bisecting varlist.json again ...

ufladrich commented 5 years ago

Here is, as a first indication, a list of variables that are present in ScenarioMIP-ssp126 but not in CMIP-piControl:

Maybe it's one of them.

treerink commented 5 years ago

My first guess: Lmon tsl

ufladrich commented 5 years ago

10 points for your guess! Removing tsl saves the run for me.

ufladrich commented 5 years ago

Here are the logs from an tsl-only run. ece2cmor-job-2341928-167.log t621-ifs-167-20190328143135.log

ufladrich commented 5 years ago

The issue with mrsol isn't solved either, unfortunately. Logs for varlist-ifs-mrsol.json: ece2cmor-job-2342925-2.log b601-ifs-002-20190328181641.log

ufladrich commented 5 years ago

On the positive side: I haven't encountered more variables than mrsol and tsl in my tests. So for now, it is these two (and zhalfo, for other reasons #388).

goord commented 5 years ago

Hi Uwe, can you test again with the branch mrsol-sdepth1?

oloapinivad commented 5 years ago

Hi all, I am back with my reporting - sorry for that :-) .

For the records, I am now testing the cmorization for the AOGCM historical, and I encounter the same error Uwe reported for mrsol.

2019-04-02 10:52:48 INFO:ece2cmor3.postproc: Post-processing target mrsol in table Emon from file /scratch/ms/it/ccpd/tmp_cmor/cj02_1850_13018/ifs_1850/CMIP6/cj02-ifs-1850/42.128.111_41.128.111_40.128.111_39.128.111.3 with cdo command -setgridtype,regular -setcode,11 -monmean -aexpr,'var39=70*var39;var40=210*var40;var41=720*var41;var42=1890*var42' -selcode,39,40,41,42
Traceback (most recent call last):
  File "/perm/ms/it/ccpd/ecearth3/cmorization/ece2cmor3/ece2cmor3/ece2cmor.py", line 153, in <module>
    main()
  File "/perm/ms/it/ccpd/ecearth3/cmorization/ece2cmor3/ece2cmor3/ece2cmor.py", line 136, in main
    cdothreads=args.ncdo)
  File "/lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/ece2cmor3/lib/python2.7/site-packages/ece2cmor3-0.9.0-py2.7.egg/ece2cmor3/ece2cmorlib.py", line 185, in perform_ifs_tasks
    ifs2cmor.execute(ifs_tasks, nthreads=taskthreads)
  File "/lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/ece2cmor3/lib/python2.7/site-packages/ece2cmor3-0.9.0-py2.7.egg/ece2cmor3/ifs2cmor.py", line 189, in execute
    pool.map(cmor_worker, tasks)
  File "/lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/ece2cmor3/lib/python2.7/multiprocessing/pool.py", line 253, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/ece2cmor3/lib/python2.7/multiprocessing/pool.py", line 572, in get
    raise self._value
IndexError: list index out of range
goord commented 5 years ago

Hi @oloapinivad and @ufladrich can you test mrsol one more in the mrsol-sdepth1 branch of ece2cmor3?

oloapinivad commented 5 years ago

Hi @goord , just tried.

I get this error right at the beginning

2019-04-02 11:50:06 ERROR:ece2cmor3.postproc: Unsupported combination of frequency monC with time operators [u'mean within years', u'mean over years'] encountered
2019-04-02 11:50:06 INFO:ece2cmor3.postproc: Post-processing target ps in table Amon from file /scratch/ms/it/ccpd/tmp_cmor/cj02_1850_10278/ifs_1850/CMIP6/cj02-ifs-1850/134.128.1.3 with cdo command -setgridtype,regular -monmean -selcode,134
2019-04-02 11:50:10 INFO:ece2cmor3.postproc: Post-processing target ps in table 3hr from file /scratch/ms/it/ccpd/tmp_cmor/cj02_1850_10278/ifs_1850/CMIP6/cj02-ifs-1850/134.128.1.3 with cdo command -setgridtype,regular -selhour,0,3,6,9,12,15,18,21 -selcode,134
2019-04-02 11:50:20 INFO:ece2cmor3.postproc: Post-processing target ps in table 6hrLev from file /scratch/ms/it/ccpd/tmp_cmor/cj02_1850_10278/ifs_1850/CMIP6/cj02-ifs-1850/134.128.1.3 with cdo command -setgridtype,regular -selhour,0,6,12,18 -selcode,134
Traceback (most recent call last):
  File "/perm/ms/it/ccpd/ecearth3/cmorization/ece2cmor3/ece2cmor3/ece2cmor.py", line 153, in <module>
    main()
  File "/perm/ms/it/ccpd/ecearth3/cmorization/ece2cmor3/ece2cmor3/ece2cmor.py", line 136, in main
    cdothreads=args.ncdo)
  File "/lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/ece2cmor3/lib/python2.7/site-packages/ece2cmor3-0.9.0-py2.7.egg/ece2cmor3/ece2cmorlib.py", line 185, in perform_ifs_tasks
    ifs2cmor.execute(ifs_tasks, nthreads=taskthreads)
  File "/lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/ece2cmor3/lib/python2.7/site-packages/ece2cmor3-0.9.0-py2.7.egg/ece2cmor3/ifs2cmor.py", line 181, in execute
    read_mask(task.target.variable, getattr(task, cmor_task.output_path_key))
AttributeError: 'cmor_task' object has no attribute 'path'
ufladrich commented 5 years ago

On 02/04/2019 13:31, Gijs van den Oord wrote:

Hi @oloapinivad https://github.com/oloapinivad and @ufladrich https://github.com/ufladrich can you test mrsol one more in the |mrsol-sdepth1| branch of ece2cmor3?

I can try, but not right now unfortunately. I'll come back as soon as I get the chance. Thanks for working on it!

treerink commented 5 years ago

I confirm the error reported by Paolo above when using the latest mrsol-sdepth1 branch.

goord commented 5 years ago

Ok @treerink I hope to have fixed this in the latest commit in the branch...

treerink commented 5 years ago

Ok, I am testing the update.

treerink commented 5 years ago

Yes mrsol successfully cmorised. I am now testing tsl.

treerink commented 5 years ago

Also tsl is successfully cmorised.

oloapinivad commented 5 years ago

I can confirm now I can cmorize all the ifs variables in cmip6-data-request-varlist-CMIP-historical-EC-EARTH-AOGCM.json without any error! Great, thanks.

treerink commented 5 years ago

The branch with the fixes is merged into the master.

So closing the issue.