search

EC-Earth / ece2cmor3

Post-processing and cmorization of ec-earth output
Apache License 2.0
13 stars 6 forks source link

missing first time step for new experiment #501

Closed oloapinivad closed 5 years ago

oloapinivad commented 5 years ago

We have just realised (following the telco discussion) that at the first time step of our simulation (historical AOGCM) we have some instantaneous variables missing. This is actually reported by QA-DKRZ (but it is not showing at which record). 

      {
            "DRS_6": [ "CF3hr", "3hr", "6hrLev", "6hrPlevPt" ],
            "DRS_7": [ "ts", "mrsos", "huss", "tas", "vas", "ps", "uas", "tslsi", "rv850", "tsl", "ta", "ua", "zg500", "psl", "va", "zg" ],
            "DRS_8": [ "gr" ],
            "annotation": "Data record totally with _FillValue.",
            "tag": "R100",
            "severity": "Warning"
        },

However, manual inspection for instance at 3hr tas shows:


cdo info tas_3hr_EC-Earth3_historical_r4i1p1f1_gr_185001010000-185012312100.nc | more
    -1 :       Date     Time   Level Gridsize    Miss :     Minimum        Mean     Maximum : Parameter ID
     1 : 1850-01-01 00:00:00       2   131072  131072 :                     nan             : -1            
     2 : 1850-01-01 03:00:00       2   131072       0 :      220.35      275.75      312.88 : -1            
     3 : 1850-01-01 06:00:00       2   131072       0 :      219.88      275.97      315.45 : -1            
     4 : 1850-01-01 09:00:00       2   131072       0 :      221.66      276.24      315.31 : -1            
     5 : 1850-01-01 12:00:00       2   131072       0 :      222.24      276.36      310.76 : -1            
     6 : 1850-01-01 15:00:00       2   131072       0 :      222.50      276.29      310.30 : -1            
     7 : 1850-01-01 18:00:00       2   131072       0 :      223.04      276.11      310.00 : -1            
     8 : 1850-01-01 21:00:00       2   131072       0 :      222.29      275.91      310.58 : -1            
     9 : 1850-01-02 00:00:00       2   131072       0 :      221.45      275.80      308.15 : -1

The hypothesis we have is that ece2cmor3 looks for the 00000 files (which is currently correctly read in our case, since I have no problem with LSM) but does not find the December file since this is the first leg of the simulation and such file does not exists (in this case it would be the 184912). Therefore misses all these above-mentioned variables which are in the 00000: even if they are identical to the IC they should be present in the cmorized data.

One possibility is to link twice the 00000 file, once as it is and once as 184912. I will test it asap.

Does other people see the same problem?

oloapinivad commented 5 years ago

I tried the above-mentioned approach but I am still getting:

2019-06-20 14:45:00 WARNING:ece2cmor3.ifs2cmor: For variable tas in table 3hr, no valid time point could be found at 1850-01-01 00:00:00...inserting missing values

My current directory structures is based on links and it is made like this:

data_1849:
totale 8
lrwxrwxrwx 1 ccpd it 82 20 giu 14.27 ICMGGchis184912 -> /lus/snx11062/scratch/ms/it/ccpd/ece3/chis/output/Output_1850/IFS/ICMGGchis+000000
lrwxrwxrwx 1 ccpd it 82 20 giu 14.27 ICMSHchis184912 -> /lus/snx11062/scratch/ms/it/ccpd/ece3/chis/output/Output_1850/IFS/ICMSHchis+000000

data_1850:
totale 104
lrwxrwxrwx 1 ccpd it 104 20 giu 14.27 ICMGGchis+000000 -> /lus/snx11062/scratch/ms/it/ccpd/ece3/chis/output/Output_1850/IFS/../../Output_1850/IFS/ICMGGchis+000000
lrwxrwxrwx 1 ccpd it  82 20 giu 14.27 ICMGGchis+185001 -> /lus/snx11062/scratch/ms/it/ccpd/ece3/chis/output/Output_1850/IFS/ICMGGchis+185001
lrwxrwxrwx 1 ccpd it  82 20 giu 14.27 ICMGGchis+185002 -> /lus/snx11062/scratch/ms/it/ccpd/ece3/chis/output/Output_1850/IFS/ICMGGchis+185002
lrwxrwxrwx 1 ccpd it  82 20 giu 14.27 ICMGGchis+185003 -> /lus/snx11062/scratch/ms/it/ccpd/ece3/chis/output/Output_1850/IFS/ICMGGchis+185003
lrwxrwxrwx 1 ccpd it  82 20 giu 14.27 ICMGGchis+185004 -> /lus/snx11062/scratch/ms/it/ccpd/ece3/chis/output/Output_1850/IFS/ICMGGchis+185004
lrwxrwxrwx 1 ccpd it  82 20 giu 14.27 ICMGGchis+185005 -> /lus/snx11062/scratch/ms/it/ccpd/ece3/chis/output/Output_1850/IFS/ICMGGchis+185005
lrwxrwxrwx 1 ccpd it  82 20 giu 14.27 ICMGGchis+185006 -> /lus/snx11062/scratch/ms/it/ccpd/ece3/chis/output/Output_1850/IFS/ICMGGchis+185006
lrwxrwxrwx 1 ccpd it  82 20 giu 14.27 ICMGGchis+185007 -> /lus/snx11062/scratch/ms/it/ccpd/ece3/chis/output/Output_1850/IFS/ICMGGchis+185007
lrwxrwxrwx 1 ccpd it  82 20 giu 14.27 ICMGGchis+185008 -> /lus/snx11062/scratch/ms/it/ccpd/ece3/chis/output/Output_1850/IFS/ICMGGchis+185008
lrwxrwxrwx 1 ccpd it  82 20 giu 14.27 ICMGGchis+185009 -> /lus/snx11062/scratch/ms/it/ccpd/ece3/chis/output/Output_1850/IFS/ICMGGchis+185009
lrwxrwxrwx 1 ccpd it  82 20 giu 14.27 ICMGGchis+185010 -> /lus/snx11062/scratch/ms/it/ccpd/ece3/chis/output/Output_1850/IFS/ICMGGchis+185010
lrwxrwxrwx 1 ccpd it  82 20 giu 14.27 ICMGGchis+185011 -> /lus/snx11062/scratch/ms/it/ccpd/ece3/chis/output/Output_1850/IFS/ICMGGchis+185011
lrwxrwxrwx 1 ccpd it  82 20 giu 14.27 ICMGGchis+185012 -> /lus/snx11062/scratch/ms/it/ccpd/ece3/chis/output/Output_1850/IFS/ICMGGchis+185012
lrwxrwxrwx 1 ccpd it 104 20 giu 14.27 ICMSHchis+000000 -> /lus/snx11062/scratch/ms/it/ccpd/ece3/chis/output/Output_1850/IFS/../../Output_1850/IFS/ICMSHchis+000000
lrwxrwxrwx 1 ccpd it  82 20 giu 14.27 ICMSHchis+185001 -> /lus/snx11062/scratch/ms/it/ccpd/ece3/chis/output/Output_1850/IFS/ICMSHchis+185001
lrwxrwxrwx 1 ccpd it  82 20 giu 14.27 ICMSHchis+185002 -> /lus/snx11062/scratch/ms/it/ccpd/ece3/chis/output/Output_1850/IFS/ICMSHchis+185002
lrwxrwxrwx 1 ccpd it  82 20 giu 14.27 ICMSHchis+185003 -> /lus/snx11062/scratch/ms/it/ccpd/ece3/chis/output/Output_1850/IFS/ICMSHchis+185003
lrwxrwxrwx 1 ccpd it  82 20 giu 14.27 ICMSHchis+185004 -> /lus/snx11062/scratch/ms/it/ccpd/ece3/chis/output/Output_1850/IFS/ICMSHchis+185004
lrwxrwxrwx 1 ccpd it  82 20 giu 14.27 ICMSHchis+185005 -> /lus/snx11062/scratch/ms/it/ccpd/ece3/chis/output/Output_1850/IFS/ICMSHchis+185005
lrwxrwxrwx 1 ccpd it  82 20 giu 14.27 ICMSHchis+185006 -> /lus/snx11062/scratch/ms/it/ccpd/ece3/chis/output/Output_1850/IFS/ICMSHchis+185006
lrwxrwxrwx 1 ccpd it  82 20 giu 14.27 ICMSHchis+185007 -> /lus/snx11062/scratch/ms/it/ccpd/ece3/chis/output/Output_1850/IFS/ICMSHchis+185007
lrwxrwxrwx 1 ccpd it  82 20 giu 14.27 ICMSHchis+185008 -> /lus/snx11062/scratch/ms/it/ccpd/ece3/chis/output/Output_1850/IFS/ICMSHchis+185008
lrwxrwxrwx 1 ccpd it  82 20 giu 14.27 ICMSHchis+185009 -> /lus/snx11062/scratch/ms/it/ccpd/ece3/chis/output/Output_1850/IFS/ICMSHchis+185009
lrwxrwxrwx 1 ccpd it  82 20 giu 14.27 ICMSHchis+185010 -> /lus/snx11062/scratch/ms/it/ccpd/ece3/chis/output/Output_1850/IFS/ICMSHchis+185010
lrwxrwxrwx 1 ccpd it  82 20 giu 14.27 ICMSHchis+185011 -> /lus/snx11062/scratch/ms/it/ccpd/ece3/chis/output/Output_1850/IFS/ICMSHchis+185011
lrwxrwxrwx 1 ccpd it  82 20 giu 14.27 ICMSHchis+185012 -> /lus/snx11062/scratch/ms/it/ccpd/ece3/chis/output/Output_1850/IFS/ICMSHchis+185012

The directory provided to ece2cmor3 is the data_1850

goord commented 5 years ago

Hi Paolo, this appears to be a bug in the code: where ece2cmor3 is supposed to find ICMGGchis+000000 for the initial step instantaneous fields, it cannot find the file (due to a bug) and inserts missing values. I am fixing this now. As a consequence, all runs that have not linked the ICMGGexpn+000000 under the name ICMGGexpn+184912 have missing values in their initial time step.

goord commented 5 years ago

Hi Paolo, I pushed the fix to the master branch, so you can test your first time step.

oloapinivad commented 5 years ago

Hi @goord the problem is solved for all the IFS variables. I have some reports still by QA-DKRZ from NEMO variables but I am trying to understand: if it is the case this would be a different issue. Thanks