problem with variable mrsow in LS3MIP experiments

francocatalano commented 4 years ago

The ece2cmor computation of LS3MIP variable mrsow requires constant field soil type (43.128). The LS3MIP experiments have been run saving this variable only in the in the +000000 ICMGG output file. For some reason, it looks like ece2cmor3 expects to find field 43.128 in all the output ICMGG files. This gives the following error when runing ece2cmor3: 2020-02-13 11:21:18 ERROR:ece2cmor3.grib_filter: Field missing in the first day of file: code 43.128, level type 1, level 0. Dismissing task mrsow in table Eday In order to solve this problem ece2cmor3 has to be modified to read constant field 43.128 only once from ICMGG+000000 file. @treerink @goord

goord commented 4 years ago

If I then process the filtered grib file with exactly the same cdo command, I get a correct netcdf file which makes the bug hard to pinpoint

francocatalano commented 4 years ago

what is very strange (and worrisome!) is that the code is not displaying a deterministic behavior. Could it be that the cdo command called from python uses, for some reason, a different version of cdo?

goord commented 4 years ago

Intermediate findings:

The grib files leading to hfdsn_LImon_*.nc all seem to be correct, and running cdo manually on the cca head node on these flies produces correct nc files.
ece2cmor3 in single-threaded mode produces the correct files.

I am now running ece2cmor3 with less threads on cca

goord commented 4 years ago

Hi @francocatalano when I run with --npp 10, but I allocate 18 threads on cca like you I don't encounter any errors. It could be that if you run with the maximal 18 threads cdo somehow goes out of memory (silently apparently) during the processing of hfdsn_LImon.nc. I suggest you either run ece2cmor3 on a parallel queue node with 18 threads or in the fractional queue with 10 threads or less. In the latter case make sure to allocate still 18 cores.

The issue is a bit disturbing as we get no messages whatsoever from cdo it failed to process a variable...

francocatalano commented 4 years ago

Hi @goord Thanks. I'll give it a try. I can get all the latest fixes by checking out master, right?

goord commented 4 years ago

Yes

francocatalano commented 4 years ago

Hi @goord I've just tried (after having switched to master branch of ece2cmor3) your suggestion of setting npp to 10. I have launched 50 legs at once. Four of them give the following error:

2020-04-08 09:18:17 ERROR:ece2cmor3.cdoapi: (returncode:1) cdo(2) monmean: Process started cdo(3) expr: Process started cdo(4) selcode: Process started Warning (cgribexScanTimestep1): Record 5 (id=141.128 lev1=0 lev2=0) timestep 1: Inconsistent verification time! Error (grb_read_record): Failed to read GRIB record

I don't think testing a further reduction of npp would be the solution. Most probably it would work, by chance, on a number of legs and still give errors on few of them.

treerink commented 4 years ago

@goord here what you asked for, they are results of several of recent tests:

find . -name hfdsn_LImon_*.nc|sort

./cmor-cmip-test-all-1/t001/ifs/001/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/LImon/hfdsn/gr/v20200318/hfdsn_LImon_EC-Earth3_piControl_r1i1p1f1_gr_199001-199012.nc
./cmor-cmip-test-all-2/t002/ifs/001/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/LImon/hfdsn/gr/v20200320/hfdsn_LImon_EC-Earth3_piControl_r1i1p1f1_gr_199001-199012.nc
./cmor-cmip-test-all-3/t002/ifs/001/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/LImon/hfdsn/gr/v20200326/hfdsn_LImon_EC-Earth3_piControl_r1i1p1f1_gr_199001-199012.nc
./cmor-cmip-test-all-3/t002/ifs/002/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/LImon/hfdsn/gr/v20200326/hfdsn_LImon_EC-Earth3_piControl_r1i1p1f1_gr_199101-199112.nc
./cmor-cmip-test-all-3/t002/ifs/003/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/LImon/hfdsn/gr/v20200326/hfdsn_LImon_EC-Earth3_piControl_r1i1p1f1_gr_199201-199212.nc
./cmor-cmip-test-all-3/t002/ifs/004/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/LImon/hfdsn/gr/v20200326/hfdsn_LImon_EC-Earth3_piControl_r1i1p1f1_gr_199301-199312.nc
./cmor-cmip-test-all-3/t002/ifs/005/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/LImon/hfdsn/gr/v20200326/hfdsn_LImon_EC-Earth3_piControl_r1i1p1f1_gr_199401-199412.nc
./cmor-cmip-test-all-3/t002/ifs/006/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/LImon/hfdsn/gr/v20200326/hfdsn_LImon_EC-Earth3_piControl_r1i1p1f1_gr_199501-199512.nc
./cmor-cmip-test-all-3/t002/ifs/007/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/LImon/hfdsn/gr/v20200326/hfdsn_LImon_EC-Earth3_piControl_r1i1p1f1_gr_199601-199612.nc
./cmor-cmip-test-all-3/t002/ifs/008/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/LImon/hfdsn/gr/v20200326/hfdsn_LImon_EC-Earth3_piControl_r1i1p1f1_gr_199701-199712.nc
./cmor-cmip-test-all-3/t002/ifs/009/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/LImon/hfdsn/gr/v20200326/hfdsn_LImon_EC-Earth3_piControl_r1i1p1f1_gr_199801-199812.nc
./cmor-cmip-test-all-3/t002/ifs/010/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/LImon/hfdsn/gr/v20200326/hfdsn_LImon_EC-Earth3_piControl_r1i1p1f1_gr_199901-199912.nc
./cmor-cmip-test-all-4/t002/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/LImon/hfdsn/gr/v20200326/hfdsn_LImon_EC-Earth3_piControl_r1i1p1f1_gr_199001-199012.nc
./cmor-cmip-test-all-4/t002/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/LImon/hfdsn/gr/v20200326/hfdsn_LImon_EC-Earth3_piControl_r1i1p1f1_gr_199101-199112.nc
./cmor-cmip-test-all-4/t002/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/LImon/hfdsn/gr/v20200326/hfdsn_LImon_EC-Earth3_piControl_r1i1p1f1_gr_199201-199212.nc
./cmor-cmip-test-all-4/t002/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/LImon/hfdsn/gr/v20200326/hfdsn_LImon_EC-Earth3_piControl_r1i1p1f1_gr_199301-199312.nc
./cmor-cmip-test-all-4/t002/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/LImon/hfdsn/gr/v20200326/hfdsn_LImon_EC-Earth3_piControl_r1i1p1f1_gr_199501-199512.nc
./cmor-cmip-test-all-4/t002/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/LImon/hfdsn/gr/v20200326/hfdsn_LImon_EC-Earth3_piControl_r1i1p1f1_gr_199701-199712.nc
./cmor-cmip-test-all-4/t002/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/LImon/hfdsn/gr/v20200326/hfdsn_LImon_EC-Earth3_piControl_r1i1p1f1_gr_199801-199812.nc
./cmor-cmip-test-all-4/t002/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/LImon/hfdsn/gr/v20200326/hfdsn_LImon_EC-Earth3_piControl_r1i1p1f1_gr_199901-199912.nc
./cmor-cmip-test-all-5/t002/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/LImon/hfdsn/gr/v20200402/hfdsn_LImon_EC-Earth3_piControl_r1i1p1f1_gr_199001-199101.nc

It looks like that only in the most recent one december is not reached?

goord commented 4 years ago

Hi @francocatalano , do you still have the temporary files belonging to those failed legs? I want to have a look at the filtered grib files containing snow depth (141). Can you verify that for the failed legs hfdsnLImon*.nc is missing?

Sorry to bother you like that, this is really a nasty bug popping up once in a while. If you want to proceed you probably can re-cmorize the failed 5 legs and be done with it, but from my perspective it is essential to understand what's going on

francocatalano commented 4 years ago

Hi @goord you can find all the temp files here: /scratch/ms/it/ccfc/cmorisation/temp-cmor-dir The failed legs are: 014, 017, 024, 031 I have opened read permissions but let me know in case of problems.

I don't find very useful for me to proceed now launching again the failed jobs. The code is still performing not deterministically with random errors. Please let me know as soon as the bug is completely fixed. And, of course, if I can help with more info. Thanks a lot for your effort.

francocatalano commented 4 years ago

Can you verify that for the failed legs hfdsnLImon*.nc is missing?

@goord I confirm. hfdsn is missing for all the failed legs.

goord commented 4 years ago

Hi Franco, can you make your experiment output readable for me (now I can only access leg 1 & 2) so I can launch a large number of simultaneous (yet distinct) jobs myself? thx

francocatalano commented 4 years ago

Hi @goord Done.

goord commented 4 years ago

Franco I notice the output is missing 0-hour fields on january 1st for every year, same problem as #612

francocatalano commented 4 years ago

Hi Franco I noticed you have ICM{GG,SH}ECE3+198012 in your leg 2 directory containing copies of fields at 1981-1-1 00:00:00. This confuses ece2cmor3, it will think that this is the actual first file of the leg. You can remove those files and the cmorization should then work properly.

Hi @goord You recommended removing these files. I can copy again them from ECFS if needed. Let me know.

goord commented 4 years ago

Ah I thought those were copied timestamps... Could you append each of these to the december gridpoint file of the previous year?

francocatalano commented 4 years ago

@goord I am not sure I understood what you asked for. What I can do is to copy the ICM{GG,SH}ECE3+{yyyy-1}12 in the output leg folders. The timestamp of these files is yyyy-01-01 00:00:00 I have copied back the 002 files. Could you please check and confirm these are the files needed. Once you confirm I will proceed restoring these files for all the legs. Thanks.

goord commented 4 years ago

Hi @francocatalano the problem is that ece2cmor3 may get confused with the current setup because it will find the ICMGGECE3+yyyy12 files twice. I have programmed ece2cmor3 such that it will look for those files anywhere in the output/ifs directory (it moves one directory level up) to get the zero-hour 01/01 fields and this search will give back 2 results now.

One option would be to run ece2cmor3 in a symlink of the leg with an extra level, e.g.

work/002/output -> ECE3/output/ifs/002

Another option would be to append the 'previous' december grib files for leg x to the actual december grib files of leg x-1

francocatalano commented 4 years ago

Hi @goord. I think that manipulating the grib files can be difficult (cdo cannot work with ICMGG files) and dangerous too since there is the risk of introducing changes in the file structure that may lead to additional problems. Therefore, I would go with the other alternative. May it help if I restore the ICM{GG,SH}ECE3+{yyyy-1}12 in a subfolder, let's say ECE3/output/ifs/002/prev_year ?

goord commented 4 years ago

Hi @francocatalano it's the other way around: the December file of the previous leg has to become invisible because it contains no useful information. Wait, I'll create a branch to let ece2cmor3 handle this situation automatically...

francocatalano commented 4 years ago

Hi @francocatalano it's the other way around: the December file of the previous leg has to become invisible because it contains no useful information. Wait, I'll create a branch to let ece2cmor3 handle this situation automatically...

ok. Thanks.

goord commented 4 years ago

Hi Franco, fixing this in ece2cmor3 appears more complicated than I anticipated but the trick with the links should work if you restore all those december files from ecfs, you just include

printf -v CURLEG "%03d" $LEG
printf -v PRVLEG "%03d" $((LEG-1))
WORKDIR=$SCRATCH/ls3mip/work/$PBS_JOBID
mkdir -p $WORKDIR/$CURLEG
mkdir -p $WORKDIR/$PRVLEG
SRCDIR=/scratch/ms/it/ccfc/ECEARTH-RUNS_ls3mip_pdLC_ssp126/ECE3/output/ifs/${LEG}
for f in $SRCDIR/ICM{GG,SH}ECE3+$((1980+LEG-1))?? ; do
    ln -s $f $WORKDIR/$CURLEG/$(basename $f)
done
for f in $SRCDIR/ICM{GG,SH}ECE3+$((1980+LEG-2))12 ; do
    ln -s $f $WORKDIR/$PRVLEG/$(basename $f)
done
for f in /scratch/ms/it/ccfc/ECEARTH-RUNS_ls3mip_pdLC_ssp126/ECE3/output/ifs/001/ICM{GG,SH}ECE3+000000 ; do
    ln -s $f $WORKDIR/$(basename $f)
done

in the submission and let ece2cmor3 run on $WORKDIR/$CURLEG instead of the original directory

francocatalano commented 4 years ago

for f in $SRCDIR/ICM{GG,SH}ECE3+$((1980+LEG-2))?? ; do
    ln -s $f $WORKDIR/$PRVLEG/$(basename $f)
done

@goord for LEG 001 ECE3+$((1980+LEG-2))12 do not exist and, therefore, $PRVLEG is empty. ece2cmor3 must take into account this

francocatalano commented 4 years ago

for f in $SRCDIR/ICM{GG,SH}ECE3+$((1980+LEG-1))?? ; do
    ln -s $f $WORKDIR/$CURLEG/$(basename $f)
done
for f in $SRCDIR/ICM{GG,SH}ECE3+$((1980+LEG-2))12 ; do
    ln -s $f $WORKDIR/$PRVLEG/$(basename $f)
done

further: with the above lines of code the job for LEG+1 tries to overwrite the symlinks created by the job for LEG. The problem is, considering for example LEGs 003 and 004: ICMGGECE3+198212 in /scratch/ms/it/ccfc/ECEARTH-RUNS_ls3mip_pdLC_ssp126/ECE3/output/ifs/004 has the same name of ICMGGECE3+198212 in /scratch/ms/it/ccfc/ECEARTH-RUNS_ls3mip_pdLC_ssp126/ECE3/output/ifs/003 but they are NOT the same file. Actually, the former contains only one time step while the latter contains all december timesteps. Therefore, the above cannot work

francocatalano commented 4 years ago

ah, ok. I have to put that inside the pbs job in order that PBS_JOBID is defined

francocatalano commented 4 years ago

@goord I've tried to do that but still $PBS_JOBID is not defined. I attach the submission script. Please have a look at it and let me know. Thanks. submit-at-cca-ece2cmor-leg-job-pdLC-ssp126_fix.sh.txt

goord commented 4 years ago

@francocatalano I didn' realize you were using Thomas' job script generator. The job id is just there to create a unique directory per job, but you can use the leg for that too, e.g.

WORKDIR='$running_directory/temp_work/ifs_$CURLEG'

Then the script should create directories

temp_work
    |-ifs_002
        |-001
        |-002
    |-ifs_003
        |-002
        |-003
    |...

in which ece2cmor3 runs and can only see the correct symlinks

francocatalano commented 4 years ago

@goord The latest fixes are still on the master branch, right?

goord commented 4 years ago

Right

francocatalano commented 4 years ago

Hi @goord I am still getting random errors of the kind: 2020-04-10 09:46:13 ERROR:ece2cmor3.cdoapi: (returncode:1) cdo(2) daymean: Process started cdo(3) expr: Process started cdo(4) selcode: Process started Error (grb_read_record): Failed to read GRIB record

and:

2020-04-10 10:03:47 ERROR:ece2cmor3.cdoapi: (returncode:1) cdo(2) monmean: Process started cdo(3) expr: Process started cdo(4) selcode: Process started Error (grb_read_record): Failed to read GRIB record

Looking at the files, hfdsn_*.nc is still missing for Eday (first kind of error) or LImon (second kind of error) Here is the submit script: submit-at-cca-ece2cmor-leg-job-pdLC-ssp126_fix.sh.txt

and here the logfiles: ECE3-ifs_010-010-20200410093728.cmor.log ECE3-ifs_010-010-20200410093728.log ECE3-ifs_040-040-20200410095332.cmor.log ECE3-ifs_040-040-20200410095332.log ECE3-ifs_043-043-20200410100439.cmor.log ECE3-ifs_043-043-20200410100439.log

Additionally, I have noticed that the logfiles have been renamed and now include twice the leg number, which prevents them to be correctly moved to cmorised-results/cmor-ls3mip-cmip/ECE3/logs/.

francocatalano commented 4 years ago

wait...I've just realized that did not update the nested cmor tables. Running the test again.

francocatalano commented 4 years ago

@goord Unfortunately, the test with the updated nested cmor tables is giving the same kind of errors, although for different legs. Still behaving randomly.

goord commented 4 years ago

Hi @francocatalano could you change the permissions to the restored ICM{SH,GG}ECE3+(yyyy-1)12 files as well? Thanks

francocatalano commented 4 years ago

@goord Ops...I overlooked that. Now it's done.

goord commented 4 years ago

Hi Franco, I figured out you do have to concatenate the december files to get the correct results with ece2cmor3, because now the fluxes that been accumulated in the last 3 hours of december will be in the next leg and therefore not noticed by ece2cmor3.

I advise you to move all december files to a backup location and the single time-slice files to somewhere else, and then use the linux cat:

cat backup/ICMGGECE3+198012 backup2/ICMGGECE3+198012 > 001/ICMGGECE3+198012
cat backup/ICMSHECE3+198012 backup2/ICMSHECE3+198012 > 001/ICMSHECE3+198012

to produce the normal EC-Earth output form. Then update ece2cmor3 to the latest master and test the cmorization. I have tested on cca and got still some errors in hfdsnLImon* with too few time stamps, but at least the rest of your data will be ok.

francocatalano commented 4 years ago

Hi @goord I have updated ece2cmor3 and the nested cmor tables and done a test following your latest suggestion. Leg 001 was processed correctly but I got the following errors for the other legs: 2020-04-14 17:35:45 ERROR:ece2cmor3.grib_filter: Assumed previous month file for /scratch/ms/it/ccfc/cmorisation//temp_work/ifs_002/002/ICMSHECE3+198101: /scratch/ms/it/ccfc/cmorisation/temp_work/ifs_002/ICMSHECE3+000000, this is probably not correct!

and: 2020-04-14 17:44:45 ERROR:ece2cmor3.ifs2cmor: CMOR failed to load table Amon, the following variable will be skipped: hfls. Reason: Problem with 'cmor.load_table'. Please check the logfile (if defined).

and in the .cmor.log files: ! Error: approximate time axis interval is defined as 86400.000000 seconds (1.000000 days), for value 1 we got a difference of 15854400.000000 seconds (183.500000 days), which is 18250.000000 % , seems too big, check your values

These are the logfiles for leg 002: ECE3-ifs_002-002-20200414173544.cmor.log ECE3-ifs_002-002-20200414173544.log

This is becoming very frustrating. I attach my submit script: submit-at-cca-ece2cmor-leg-job-pdLC-ssp126_fix.sh.txt

Please, revise it and let me know whether I misinterpreted somehow what you suggested. Thanks.

goord commented 4 years ago

Hi Franco, sorry you can remove the ifs_$CURLEG inner subdirectory from your WORKDIR, in this way you mimic the normal output structure of ifs such that ece2cmor3 can find the correct December files in the previous leg directory.

BTW you can create all those concatenated files and links once.

treerink commented 4 years ago

Hi, I want to close this issue, the mrsow issue is solved. For follow up questions or assistance for handling the data during cmorisation please open a new issue. Also further discussion of hfdsn LImon should be continued in a separate issue if necessary. Currently it looks like we have to accept that at cca we sometimes have a glitch with hfdsn LImon and that just rerunning ece2cmor3 will fix it.

francocatalano commented 4 years ago

Hi @goord I have done that. Now, apparently, there are no errors reported in the cmorization logfiles. Nevertheless, some of the legs still randomly produce incomplete time series for the variable hfdsnLImon*. @treerink Before closing the issue I would like at least to re-run nctime and see.

goord commented 4 years ago

Hi Franco, I noticed that as well and, to be honest, I don't know what's causing this. It may be something with memory in the fractional queue of cca, or something else. Tomorrow I will do a test in the parallel queue.

francocatalano commented 4 years ago

Hi @goord Please also note that while for some legs the incomplete time series is evident in the file name, for others it is not. That is, the file name is correct but the dimension is much smaller than the other files for the same variable. For example: -rw-r----- 1 ccfc it 1,4M 15 apr 16.55 hfdsn_LImon_EC-Earth3_amip-lfmip-pdLC_r1i1p1f1_gr_207801-207812.nc -rw-r----- 1 ccfc it 1,3M 15 apr 16.59 hfdsn_LImon_EC-Earth3_amip-lfmip-pdLC_r1i1p1f1_gr_207901-207912.nc -rw-r----- 1 ccfc it 823K 15 apr 16.58 hfdsn_LImon_EC-Earth3_amip-lfmip-pdLC_r1i1p1f1_gr_208001-208007.nc -rw-r----- 1 ccfc it 450K 15 apr 16.59 hfdsn_LImon_EC-Earth3_amip-lfmip-pdLC_r1i1p1f1_gr_208101-208112.nc -rw-r----- 1 ccfc it 1,3M 15 apr 17.03 hfdsn_LImon_EC-Earth3_amip-lfmip-pdLC_r1i1p1f1_gr_208201-208212.nc -rw-r----- 1 ccfc it 1,1M 15 apr 17.02 hfdsn_LImon_EC-Earth3_amip-lfmip-pdLC_r1i1p1f1_gr_208301-208310.nc

which makes very painful to assess whether ece2cmor3 produced the expected results or not.

Furthermore, I found out that there are incomplete time series also for variable hfdsnEday* And, to make things even more difficult, the legs with problems are not the same for LImon and Eday. Which makes practically impossible to isolate the legs to re-run.

you can have a look at the files produced with my latest test here: /scratch/ms/it/ccfc/cmorisation/cmorised-results/cmor-ls3mip-cmip/ECE3

goord commented 4 years ago

Hi Franco, I tried the parallel queue, without succes. I also noticed that the filtered grib files corresponding to hfdsn, the file named 147.128.1_146.128.1_177.128.1_176.128.1.3_141.128.1.3, is never cleaned up... this may give us a lead to the problem

goord commented 4 years ago

Hi Franco it seems that somehow the tool is mixing up hfdsl and hfdsn once in a while...

francocatalano commented 4 years ago

@goord Could it be a problem of misinterpreted wildcards in the code, somehow? Just guessing...

goord commented 4 years ago

Hi Franco, the problem is wirt simultaneous threads writing to the same file (I believe). Going to fix it asap.

goord commented 4 years ago

Hi @francocatalano I created the branch flux-inst-bugfix (the issue originates from the fact that hfdsn is a combination of fluxes and instantaneous snow depth) which may solve your problem. I ran 25 years on your data and no longer saw the missing time stamps in the hfdsn files.

To test, run git checkout flux-inst-bugfix in your ece2cmor3 repo, and then in your conda ece2cmor3 environment reinstall (i.e. python setup.py install), and resubmit your job.

@treerink it would be nice if the branch, if validated, could end up in the release

goord commented 4 years ago

BTW Franco, note that you still have to run in the directory with the concatenated december files (or create them in the submission script).

treerink commented 4 years ago

Ok, @francocatalano if you let me know whether the glitch doen not longer occur with the flux-inst-bugfix branch, then I will merge the branch and thereafter do a final test-all test before releasing #605.

francocatalano commented 4 years ago

@goord just tried to get the fix but got the following error: git checkout flux-inst-bugfix error: pathspec 'flux-inst-bugfix' did not match any file(s) known to git what's wrong?

treerink commented 4 years ago

Hi Franco, first pull in your master (or anywhere):

git pull
git checkout flux-inst-bugfix
python setup.py install          # in your ece2cmor3 root dir

EC-Earth / ece2cmor3

problem with variable mrsow in LS3MIP experiments #594