Test ocean output cmorization

The ocean cmorization needs to be tested again with the most recent version of the CMIP6 tables and the most recent nemo output. Report problems as github issues.

Testrun yields:

./ece2cmor.py /nobackup/users/oord/nemo/001 19900101 --npp 2 -o

C Traceback:
In function: cmor_outpath_exist
! called from: cmor_dataset_json
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Warning: You defined your output directory to be: './cmor', but this
! directory does not exist. CMOR will create it!
!
!!!!!!!!!!!!!!!!!!!!!!!!!

WARNING:cmor_target:Input table /usr/people/oord/src/ece2cmor3/resources/tables/CMIP6_CV_test.json has been ignored. Reason: No JSON object could be decoded
ERROR:cmor_target:Error parsing cell measures time: mean grid_longitude: mean for variable msftyrhompa in table Omon
ERROR:cmor_target:Error parsing cell measures time: mean grid_longitude: mean for variable msftyrho in table Omon
ERROR:cmor_target:Error parsing cell measures time: mean grid_longitude: mean for variable msftyyz in table Omon
ERROR:cmor_target:Error parsing cell measures time: mean grid_longitude: mean for variable msftyzmpa in table Omon
ERROR:jsonloader:Could not find cmor target for variable orog in table AERmon
ERROR:jsonloader:Could not find cmor target for variable snw in table Eday
ERROR:jsonloader:Could not find cmor target for variable mrro in table Eday
INFO:jsonloader:Found 245 cmor target variables in input variable list.
ERROR:jsonloader:Could not find parameter table entry for masscello in table Omon...skipping variable.
ERROR:jsonloader:Could not find parameter table entry for hfbasin in table Omon...skipping variable.
ERROR:jsonloader:Could not find parameter table entry for hfx in table Omon...skipping variable.
ERROR:jsonloader:Could not find parameter table entry for hfy in table Omon...skipping variable.
ERROR:jsonloader:Could not find parameter table entry for msftyrho in table Omon...skipping variable.
ERROR:jsonloader:Could not find parameter table entry for tosga in table Omon...skipping variable.
ERROR:jsonloader:Could not find parameter table entry for bigthetaoga in table Omon...skipping variable.
INFO:jsonloader:Created 238 ece2cmor tasks from input variable list.
INFO:ece2cmorlib:Selected 37 NEMO tasks from 238 input tasks
INFO:nemo2cmor:Creating NEMO grids in CMOR...
INFO:nemo2cmor:Executing 37 NEMO tasks...
INFO:nemo2cmor:Cmorizing NEMO tasks...
INFO:nemo2cmor:Loading CMOR table SIday to process 4 variables...
INFO:nemo2cmor:Creating time axes for table SIday...
INFO:nemo2cmor:Creating depth axes for table SIday...
ERROR:nemo2cmor:Could not find depth axis variable depthu in NEMO output file u; skipping depth axis creation.
ERROR:nemo2cmor:Could not find depth axis variable depthv in NEMO output file v; skipping depth axis creation.
INFO:nemo2cmor:Cmorizing source variable siconc to target variable siconc...

C Traceback:
In function: _CV_checkSourceID
! called from: cmor_setGblAttr
! called from: cmor_write
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Warning: Your input attribute "source" with value 
! "EC-Earth 3-HR" will be replaced with value 
! "EC-Earth3-HR".
! 
! 
!  See Control Vocabulary JSON file.(/usr/people/oord/src/ece2cmor3/resources/tables/CMIP6_CV.json)
! 
!
!!!!!!!!!!!!!!!!!!!!!!!!!

ERROR:nemo2cmor:The source variable sivelu could not be found in the input NEMO data
ERROR:nemo2cmor:The source variable sithic could not be found in the input NEMO data
ERROR:nemo2cmor:The source variable sivelv could not be found in the input NEMO data
INFO:nemo2cmor:Loading CMOR table SImon to process 1 variables...
INFO:nemo2cmor:Creating time axes for table SImon...
INFO:nemo2cmor:Creating depth axes for table SImon...
ERROR:nemo2cmor:The source variable precip could not be found in the input NEMO data
INFO:nemo2cmor:Loading CMOR table Oday to process 3 variables...
INFO:nemo2cmor:Creating time axes for table Oday...
INFO:nemo2cmor:Creating depth axes for table Oday...
ERROR:nemo2cmor:Could not find depth axis variable depthu in NEMO output file u; skipping depth axis creation.
ERROR:nemo2cmor:Could not find depth axis variable depthv in NEMO output file v; skipping depth axis creation.
INFO:nemo2cmor:Cmorizing source variable tossq to target variable tossq...
INFO:nemo2cmor:Cmorizing source variable tos to target variable tos...
INFO:nemo2cmor:Cmorizing source variable sos to target variable sos...
INFO:nemo2cmor:Loading CMOR table Omon to process 29 variables...
INFO:nemo2cmor:Creating time axes for table Omon...
INFO:nemo2cmor:Creating depth axes for table Omon...
INFO:nemo2cmor:Cmorizing source variable tauvo to target variable tauvo...
INFO:nemo2cmor:Cmorizing source variable vo to target variable vo...
INFO:nemo2cmor:Cmorizing source variable vomematr to target variable vmo...
INFO:nemo2cmor:Cmorizing source variable rsntds to target variable rsntds...
INFO:nemo2cmor:Cmorizing source variable tauuo to target variable tauuo...
INFO:nemo2cmor:Cmorizing source variable uo to target variable uo...
INFO:nemo2cmor:Cmorizing source variable vozomatr to target variable umo...
INFO:nemo2cmor:Cmorizing source variable pbo to target variable pbo...
/usr/people/oord/src/ece2cmor3/cmor_utils.py:243: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  array[array == missval_in] = missval_out
INFO:nemo2cmor:Cmorizing source variable zos to target variable zos...
INFO:nemo2cmor:Cmorizing source variable zossq to target variable zossq...
INFO:nemo2cmor:Cmorizing source variable thkcello to target variable thkcello...
INFO:nemo2cmor:Cmorizing source variable thetao to target variable thetao...
INFO:nemo2cmor:Cmorizing source variable tos to target variable tos...
INFO:nemo2cmor:Cmorizing source variable tossq to target variable tossq...
INFO:nemo2cmor:Cmorizing source variable so to target variable so...
INFO:nemo2cmor:Cmorizing source variable sos to target variable sos...
INFO:nemo2cmor:Cmorizing source variable mlotst to target variable mlotst...
INFO:nemo2cmor:Cmorizing source variable friver to target variable friver...
INFO:nemo2cmor:Cmorizing source variable ficeberg to target variable ficeberg2d...
INFO:nemo2cmor:Cmorizing source variable wfo to target variable wfo...
INFO:nemo2cmor:Cmorizing source variable rsdo to target variable rsdo...
ERROR:nemo2cmor:The source variable scsaltot could not be found in the input NEMO data
ERROR:nemo2cmor:The source variable scvoltot could not be found in the input NEMO data
ERROR:nemo2cmor:The source variable zfull could not be found in the input NEMO data
ERROR:nemo2cmor:The source variable qt could not be found in the input NEMO data
ERROR:nemo2cmor:The source variable zhalf could not be found in the input NEMO data
ERROR:nemo2cmor:The source variable vovematr could not be found in the input NEMO data
ERROR:nemo2cmor:The source variable scmastot could not be found in the input NEMO data
ERROR:nemo2cmor:The source variable sctemtot could not be found in the input NEMO data

I conclude that the cmorization of the ocean works in principle, but some issues need to be resolved

I just tested with a reduced list of ocean variables (just tos) and got the error "output file already exists", as reported in #31.

Hi,

After updating the code, running the following command: ./ece2cmor.py --oce --exp $EXPID --conf metadata-conf.json -o $nemo_dir ${startyear}-${startmonth}-01

fails with a message similar to Klaus's:

!!!!!!!!!!!!!!!!!!!!!!!!! ! ! Error: Output file ( ./cmor/CMIP/EC-Earth-Consortium/EC-Earth3-HR/historical/r1i1p1f1/SIday/siconc/gn/v20170614/siconc_SIday_historical_EC-Earth3-HR_r1i1p1f1_gn_19900101-19900131.nc ) already exists, ! remove file or use CMOR_REPLACE or ! CMOR_APPEND for CMOR_NETCDF_MODE value ! in cmor_setup for convenience the file ! you were trying to write has been saved ! at: ./cmor/CMIP/EC-Earth-Consortium/EC-Earth3-HR/historical/r1i1p1f1/SIday/siconc/gn/v20170614/siconc_SIday_historical_EC-Earth3-HR_r1i1p1f1_gn_19900101-19900131.nc.copy ! !!!!!!!!!!!!!!!!!!!!!!!!!

C Traceback: ! In function: cmor_close_variable !

!!!!!!!!!!!!!!!!!!!!!!!!! ! ! Error: could not rename temporary file: ./cmor/CMIP/EC-Earth-Consortium/EC-Earth3-HR/historical/r1i1p1f1/SIday/siconc/gn/v20170614/siconc_SIday_historical_EC-Earth3-HR_r1i1p1f1_gntaUiN220925.nc to final file name: ./cmor/CMIP/EC-Earth-Consortium/EC-Earth3-HR/historical/r1i1p1f1/SIday/siconc/gn/v20170614/siconc_SIday_historical_EC-Earth3-HR_r1i1p1f1_gn_19900101-19900131.nc ! !!!!!!!!!!!!!!!!!!!!!!!!!

Traceback (most recent call last): File "./ece2cmor.py", line 91, in main(sys.argv[1:]) File "./ece2cmor.py", line 88, in main ece2cmorlib.perform_nemo_tasks(args.datadir,args.exp,startdate,length) File "/esnas/scratch/Earth/pbretonn/ece2cmor/ece2cmor3/ece2cmorlib.py", line 148, in perform_nemo_tasks nemo2cmor.execute(nemo_tasks) File "/esnas/scratch/Earth/pbretonn/ece2cmor/ece2cmor3/nemo2cmor.py", line 101, in execute execute_netcdf_task(task,ds,tab_id) File "/esnas/scratch/Earth/pbretonn/ece2cmor/ece2cmor3/nemo2cmor.py", line 131, in execute_netcdf_task cmor.close(varid) File "/shared/earth/software/Python/2.7.9-foss-2015a/lib/python2.7/site-packages/CMOR-3.2-py2.7-linux-x86_64.egg/cmor/pywrapper.py", line 792, in close return _cmor.close(var_id,0,0) _cmor.CMORError: Problem with 'cmor.close'. Please check the logfile (if defined).

Passing the argument --mode replace in the ece2cmor.py call doesn't seem to have any influence. I suspect this is more a cmor issue than a ece2cmor one but I thought it was better to report it here anyway.

Have you had similar issues? Thanks,

Pierre-Antoine

Actually, it was not a CMOR issue. The cmor_mode was just not passed to CMOR. Commit https://github.com/goord/ece2cmor3/commit/1037ec79fdb3369fd44a48d535a73a423c78bdc3 solves the issue.

Hi @pabretonniere,

I was trying ece2more on cca, and got the same error I saw you got earlier, the one above
(Error: Output file ( ./cmor/CMIP6/CMIP/institute_idEC-Earth3-HR/historical/r1i1p1f1/CFday/ps/gr/v20170901/ps_CFday_EC-Earth3-HR_historical_r1i1p1f1_gr_19900101-19900130.nc ) already exists... and finishing with File "/scratch/ms/ie/dujm/p-process/pycmorenv/lib/python2.7/site-packages/CMOR-3.2-py2.7-linux-x86_64.egg/cmor/pywrapper.py", line 792, in close return _cmor.close(var_id,0,0) CMORError: Problem with 'cmor.close'. Please check the logfile (if defined).

I cannot see the file ps_CFday_EC-Earth3-HR_historical_r1i1p1f1_gr_19900101-19900130.nc. When I run it, it stalls, and need to restart terminal. You said it was a cmor issue - I was wondering have you managed to solve it? It is just I checked commit 1037ec7, I have that version of ece2morlib.py (with cmor_mode added), but still get error

Thanks

Hi @goord .

In addition to the commit you mention, I passed "--mode replace" as argument to my ece2cmor call (./ece2cmor.py --atm --exp $EXPID --conf metadata-conf-atm.json --freq $freq --mode replace --npp 16 --tabid $proj -a --tmpdir $tmpdir $output_dir_atm). Did you also do that? Else, by default, the mode is preserve.

Hi @pabretonniere and @jmcgove I usually use the append mode for ece2cmor, but I think it doesn't matter because I see always monthly files being produced. @jmcgove, are you sure the file doesn't exist? Can you remove the entire cmor/CMIP6 directory and re-run? Also make sure you do not have the variable ps twice in your varlist file.

Hi, --mode replace removed this @pabretonniere, thanks. In my varlist @goord, ps is in different categories (Amon, CFday, 3hr), once in each, I suppose that's ok?

I now have like before, parameter 248 not found and the run aborts. I checked, there is no output in tmp output_dir, as the run aborts. I tried taking 248 type params out of varlist (like for 135 params @goord), but don't see any (fraction of cloud cover, cc). @pabretonniere I noticed you also recently had a similar issue, #50, with param 9 sub surface runoff. I wonder if there's a way for the run to keep going when these parameters are not found. Thanks

STDERR:cdo showltype: Started child process "selcode,248 /scratch/ms/ie/dujm/opt/ICMGGECE3+199001 (pipe1.1)". cdo(2) selcode (Warning): Code number 248 not found! cdo(2) selcode (Abort): No variables selected!

Traceback (most recent call last): File "./ece2cmor.py", line 68, in main(sys.argv[1:]) File "./ece2cmor.py", line 63, in main maxsizegb = args.tmpsize) File "/lus/snx11062/scratch/ms/ie/dujm/p-process/ece2cmor3.git/trunk/ece2cmorlib.py", line 135, in perform_ifs_tasks ifs2cmor.execute(ifs_tasks,cleanup = cleanup) File "/lus/snx11062/scratch/ms/ie/dujm/p-process/ece2cmor3.git/trunk/ifs2cmor.py", line 138, in execute processedtasks = postprocess(taskstodo) File "/lus/snx11062/scratch/ms/ie/dujm/p-process/ece2cmor3.git/trunk/ifs2cmor.py", line 279, in postprocess tasks_done = postproc.post_process([t for t in tasks if hasattr(t,"path")],tempdir,maxsize,ifs_griddescr) File "/lus/snx11062/scratch/ms/ie/dujm/p-process/ece2cmor3.git/trunk/postproc.py", line 40, in post_process command = create_command(task,griddes) File "/lus/snx11062/scratch/ms/ie/dujm/p-process/ece2cmor3.git/trunk/postproc.py", line 98, in create_command add_level_operators(result,task) File "/lus/snx11062/scratch/ms/ie/dujm/p-process/ece2cmor3.git/trunk/postproc.py", line 270, in add_level_operators leveltypes = cdo.get_z_axes(getattr(task,"path",None),task.source.get_root_codes()[0].var_id) File "/lus/snx11062/scratch/ms/ie/dujm/p-process/ece2cmor3.git/trunk/cdoapi.py", line 178, in get_z_axes output = self.app.showltype(input = " ".join([cdo_command.make_option(seloperator,[var]),ifile])) File "/scratch/ms/ie/dujm/p-process/pycmorenv/lib/python2.7/site-packages/cdo.py", line 227, in get raise CDOException(**retvals) cdo.CDOException: (returncode:1) cdo showltype: Started child process "selcode,248 /scratch/ms/ie/dujm/opt/ICMGGECE3+199001 (pipe1.1)". cdo(2) selcode (Warning): Code number 248 not found! cdo(2) selcode (Abort): No variables selected!

EC-Earth / ece2cmor3

Test ocean output cmorization #3