search

EC-Earth / ece2cmor3

Post-processing and cmorization of ec-earth output
Apache License 2.0
13 stars 6 forks source link

"list index out of range" with PRIMAVERA tables #291

Closed oloapinivad closed 5 years ago

oloapinivad commented 5 years ago

Hi all,

I am cmorizing PRIMAVERA output with the primavera-stream2 branch, now working with PRIMAVERA tables. However, I am getting a new error even if in theory the code should not have changed. So probably it is on my side but I cannot understand the origin of the error.

Here it is:

INFO:ece2cmor3.ifs2cmor:Creating time axis using variable mrlsl...
INFO:ece2cmor3.ifs2cmor:Creating soil depth axis using variable mrlsl...
INFO:ece2cmor3.ifs2cmor:Cmorizing source variable 228 to target variable prmin...
INFO:ece2cmor3.ifs2cmor:Loading CMOR table Primday...
WARNING:ece2cmor3.ifs2cmor:Skipped removal of nonempty work directory /scratch/ms/it/ccpd/tmp_cmor/ap00_1950_14167/atm_1950_1/PRIMAVERA/ap00-ifs-195001
Traceback (most recent call last):
  File "/perm/ms/it/ccpd/ecearth3/cmorization/ece2cmor3/ece2cmor3/ece2cmor.py", line 114, in <module>
    main()
  File "/perm/ms/it/ccpd/ecearth3/cmorization/ece2cmor3/ece2cmor3/ece2cmor.py", line 102, in main
    maxsizegb=args.tmpsize)
  File "/lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/ece2cmor3/lib/python2.7/site-packages/ece2cmor3-1.0.0-py2.7.egg/ece2cmor3/ece2cmorlib.py", line 163, in perform_ifs_tasks
    ifs2cmor.execute(ifs_tasks, cleanup=cleanup, autofilter=auto_filter, nthreads=taskthreads)
  File "/lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/ece2cmor3/lib/python2.7/site-packages/ece2cmor3-1.0.0-py2.7.egg/ece2cmor3/ifs2cmor.py", line 136, in execute
    cmorize([t for t in processed_tasks if t in supported_tasks], nthreads=nthreads)
  File "/lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/ece2cmor3/lib/python2.7/site-packages/ece2cmor3-1.0.0-py2.7.egg/ece2cmor3/ifs2cmor.py", line 335, in cmorize
    pool.map(cmor_worker, [task for task in tasks if task.status not in skip_status])
  File "/lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/ece2cmor3/lib/python2.7/multiprocessing/pool.py", line 253, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/ece2cmor3/lib/python2.7/multiprocessing/pool.py", line 572, in get
    raise self._value
IndexError: list index out of range

Does anybody has any suggestion on what is going on here? Also, the batch job is not killed but hangs.

oloapinivad commented 5 years ago

Ok,

I am quite sure that the error comes from the ifs parameter table of mrsol, which points to var 124 while it should point to var 118. It should be corrected in the ifspar.json 

  {
        "source": "118.129",
        "target": "mrsol",
        "expr": "var118=merge(70*var39,210*var40,720*var41,1890*var42)",
        "masked": "land"
    },
    {
        "source": "118.129",
        "target": "mrlsl",
        "expr": "var118=merge(70*var39,210*var40,720*var41,1890*var42)",
        "masked": "land"
    },

I think this is coming from the changes in tables during PRIMAVERA project, see https://github.com/PRIMAVERA-H2020/cmip6-cmor-tables/issues/46

Of course, this is valid only for primavera-stream2 branch

goord commented 5 years ago

Hi Paolo, the error you see usually occurs when no tasks gets through the validation. You are probably running ece2cmor3 with a very small number of requested output variables?

oloapinivad commented 5 years ago

Hi Gijs,

I was running with about 30 variables, i.e. the PRIMAVERA varlist. Actually the problem was mrsol. Removing it I was able to run it, and then fixing the code to 118 solved the issue.