support for alternate runtime directory location

[etiennesky]

Hi,

In autosubmit we store the IFS results as monthly averages (MMA files), which is already covered in the ifs_monthly_mma.sh script.

However, we have a slightly different experiment management strategy.

Instead of having all runs stored in sub-folders of a common folder (as is assumed in ece3-postproc), we have one folder for each experiment and the runs are stored in sub-folders according to start_date and member, e.g. $SCRATCH/a0ts/19901001/fc0/runtime

For tests I can trick ece3-postproc by creating some symlinks inside ECE3_POSTPROC_RUNDIR=#SCRATCH/ecearth-runs

sc32051@login1: /gpfs/scratch/bsc32/bsc32051/ecearth-runs > ls -l a0ts
total 0
drwxr-xr-x 4 bsc32051 bsc32 4096 Feb 20 13:38 post
lrwxrwxrwx 1 bsc32051 bsc32   49 Feb 20 12:09 output -> ../../a0ts/19900101/fc0/runtime/output

But now I am trying to integrate this in autosubmit, without having to resort to the symlink strategy, because I would like this to work with several members.

The idea would be to have a single variable for where a single job output is stored (not where all expids are stored), the same for post and diag output.

I managed to get this working with the ALT_RUNDIR argument to hiresclim2 and ECMean (with some modifications). However, I also have to change the CLIMDIR for ECMean, and the DIR_TIME_SERIES variable for timeseries.

Everything would be easier if ECE3_POSTPROC_RUNDIR would contain the EXPID directly, instead of adding it everywhere that the variable is used. (e.g. BASERESULTS=${ECE3_POSTPROC_RUNDIR}/$expname/output)

The definitions would be by default

ECE3_POSTPROC_BASERUNDIR=$SCRATCH/ecearth-runs [...] [[ -z $ECE3_POSTPROC_RUNDIR ]] && ECE3_POSTPROC_RUNDIR=${ECE3_POSTPROC_BASERUNDIR}/$expid

and in e.g.. in master_hiresclim.sh :
export BASERESULTS=${ECE3_POSTPROC_RUNDIR}//output

But for our autosubmit case we would do ECE3_POSTPROC_BASERUNDIR=$SCRATCH/tmp ECE3_POSTPROC_RUNDIR=$SCRATCH/$EXPID/$start_date/$member/runtime

Any thoughts? thanks!

[etiennesky]

That being said, if it is ok to add some env. variables to override the defaults on the top of each script I am happy with that. Don't want to complicate things for others.

[etiennesky]

But... I still need to be able to override the location of the runtime, so ALT_RUNDIR should point to tthe rundir of the experiment, not another top-level rundir directory.

[plesager]

This is similar to what is discussed in #8 with @oloapinivad and @jhardenberg, where we wonder about the best strategy for any kind of tree structure for input to hiresclim. But you also bring the case for in/out of ECmean.

Generally speaking, we need to break out of the assumption "of having all runs stored in sub-folders of a common folder" as you noted.

Let's first get the hiresclim input fixed and flexible. I think we have to go forward with a template in the user config file, that will be parsed in _masterhiresclim.sh and then output dir are exported. Tokens could be YYYY, MM, DD, LEGNB, EXPID. And (maybe?) get rid of the ECE3_POSTPROC_BASERUNDIR as env variable.

[etiennesky]

Hi! sounds good, as long as there is a way we can add extra tokens if the default ones are not sufficient.

As long as we can easily set the location of the runtime and where to store output I will be happy.

[etiennesky]

I am unsure that a template is the right solution, why not use the eval strategy?

It would be up to the user to make sure that the variables used in the expression are all defined, and easy to add to the top of all master scripts.

e.g.ECE3_POSTPROC_RUNDIR=$SCRATCH/ecearth-runs/$exp or $SCRATCH/$exp/$sd/$member/runtime

And I think we should be able to define the expression inside the calling script, not only in the user config file.

Maybe if you would post an example of how it would look like I can understand better what you mean by "parsing a template".

Another point I would like to raise is that I think we need 3 configurable expressions

1. OUTDIR - the directory of the IFS/NEMO output - e.g. $SCRATCH/$exp/runtime/output
2. POSTDIR - where the result of hiresclim (and ECMean clim) go - e.g. $SCRATCH/$exp/runtime/ece3post
3. DIAGDIR - where the results of ECMean and timeseries go e.g. $HOME/ecearth3/diag or $SCRATCH/$exp/runtime/ece3diag

[plesager]

Eval works fine, and as an example I just fixed the case for monthly legs in 1761abb. The tricky part is that some variables in the template may not be available in the master scripts, and we need to eval only when needed and protect the path (or just part of it actually) with single quotes. Vicious part is that variables in the path to be evaluated that are only in the master script must not be single quoted.

My idea of a template is not different, just somewhat more generic, instead of eval the parsing can rely on sed, perl or somehting else. Also it takes off the burden from the end user to figure out which variable (e.g. $m for month) can be quoted without going into the code, or the required arithmetic if needed (e.g. $(printf %03d $(( (year-${yref})*12+m))) for 3-digit leg number). Bottom-line is that a template with tokens to be sed-ed is much easier on the end user.

[etiennesky]

Hi @oloapinivad I assume at ISAC you have no need for this change? Current code is fine with you?

[oloapinivad]

Hi @etiennesky I am a bit lost about the modification you propose, on my side the current version (with the introduced eval strategy) works fine.

There are minor things as lack of personalization of TMPDIR, the structure (invert Post_$year and mon in the directory tree) and the file structure (add "mon" to the filename) but these are totally minor things!

[plesager]

The last commit 5cf743d makes the OUTDIR (not used anymore, but IFSRESULTS and NEMORESULTS instead), POSTDIR, DIAGDIR and CLIMDIR really flexible. Does it fix this issue and #8?

[etiennesky]

Hi Phil thanks for merging it. I hope it is not too confusing, I started editing the README file but wanted to wait for acceptance before changing all relevant parts.

[etiennesky]

I am happy with the current implementation, probably need to adapt timeseries to close this issue