support for ISAC file structure

[oloapinivad]

This is not a real issue but it is just to keep you updated: I am introducing the support for ISAC CNR file output which is based on an enviromental variable: if the variable is not set, the script is using the standard structure, otherwise it follows our file structure.

I must introduce it in plenty of scripts in order to keep the general structure compatible to the original version. An example is here below:

if [[ -n ${ECE3_POSTPROC_ISAC_STRUCTURE} ]] ; then
    IFSRESULTS=$BASERESULTS/Output_${year}/IFS
else
    IFSRESULTS=$BASERESULTS/ifs/$(printf %03d $((year-${yref}+1)))
fi

If you have any suggestion please let me know! Also in this direction, we do not need $yref but we would like to re-introduce the possibility of running master_hiresclim.sh on multiple years. I will think for a solution. Best, Paolo

[plesager]

For the file structure, I still need to think about it. For the YREF argument, we can make it optional or, better, optional only if the ECE3_POSTPROC_ISAC_STRUCTURE is true. For the multi year functionality within _masterhiresclim.sh, I wonder why you need it. It is so easy to submit parallel jobs for each year, and the faster parallelization you can get. Which is basically what hc.sh does.

[oloapinivad]

I am discussing a bit with @jhardenberg, for the moment I am implementing a simple solution in _masterhiresclim.sh in revision 038b14d. I am wondering if we can think about something hard-coded in the config file (more portable), perhaps flexible to potential further new file structures. Also, in order to support in the best way the monthly legs or any other combination the new solution should be expanded using dynamic strings that will be evaluated inside each ifs*.sh or _nemopost.sh script. I will work on it in the next days.

I have seen now the hc.sh solution, I was not aware of it, I agree it is a smart one so we can neglect the multi-year option for _masterhiresclim.sh. I will try to update also the autopost.sh, that is a nice companion script (I was using it in order to automatically detect which years are not post-processed and run a postproc job on it).

Once I will come out with some updates I will let you know, if you find them reasonable we can try to merge them with the main fork.

Thanks! Paolo

[plesager]

The simple solution in _masterhiresclim.sh where you export the data dirs looks good. I would not worry too much about the monthly legs for now. To be exhaustive we would have to account for 1,2,3,4, and 6-month legs. This is not a priority, and we could live with only support for 1-month legs with default tree output. That said, I am always surprised with what other users are doing :)

To accommodate any other (yearly legged) output tree, I agree that a template in the config file is probably the best and easiest. It would include at least one of two tokens (leg number or/and year), and then be parsed with a call to sed for example in the _masterhiresclim.sh.

[plesager]

See #9, we need to go forward with a template in the config file. I can work on it once I merged the simple solution you laid out in 038b14d. Just ask for a pull request once you have something working for you. Unless you are already working on something similar?

[oloapinivad]

Hi, yes I have seen. My last working version is https://github.com/oloapinivad/ece3-postproc/commit/fbce67b1ab6be475d8dde2a27d05f4cf4dd8dd25, in a sort of development branch for us. I will try to create a pull request only for the hiresclim2 part (not sure if I can do it). I am using eval so it may be deprecated if we shift to the token based version mentioned in https://github.com/plesager/ece3-postproc/issues/9. However, it should have now the support for the monthly leg too.

[etiennesky]

Hi, regarding the multi year functionality within master_hiresclim.sh I agree it should remain outside of it.

[etiennesky]

Hi @oloapinivad I am trying to understand how the eval works for you, I cannot find anywhere what values you pass to $BASERESULTS, can you give an example? Thanks!

[oloapinivad]

Hi @etiennesky, sorry but I am not sure I am following! In _masterhiresclim.sh we define $BASERESULTS following the @plesager standard structure

# where to find the results from the EC-EARTH experiment if [[ -n $ALT_RUNDIR ]] then export BASERESULTS=$ALT_RUNDIR/$expname/output else export BASERESULTS=${ECE3_POSTPROC_RUNDIR}/$expname/output fi [[ ! -d $BASERESULTS ]] && echo "*EE* Experiment output dir $BASERESULTS does not exist!" && exit 1

Then, we eval few lines after the values to define IFSRESULTS and NEMORESULTS... Hope this helps, but please ask if you need more details!

Best, Paolo

[etiennesky]

Hi @oloapinivad sorry for not being clear, I men to say what values to you set ECE3_POSTPROC_RUNDIR or ALT_RUNDIR which require the eval statements?

For example this statement in itself does not seem to require being eval'd, since $year is defined in the script.

NEMORESULTS0=$BASERESULTS/Output_${year}/NEMO

Just one example of the script calling master_hiresclim.sh or value of ECE3_POSTPROC_RUNDIR which contains variables to be eval'd would help.

[oloapinivad]

Hi @etiennesky , now I see your point! Yes you are right, the statement in many cases does not need eval: the idea was to introduce a flexible solution, for instance if you use monthly legs $IFSRESULTS0 and $NEMORESULTS0 need to be evaluated inside the script (see _ifsmonthly.sh).

Indeed, my original solution did not include eval but just a couple of if/else statement (see commit https://github.com/oloapinivad/ece3-postproc/commit/038b14dd6cb2922cda459d8f4a9880886f093335).

So, I introduced eval to support for monthly legs (@plesager actually fixed my original wrong commit), and this may be flexible also in the future. But if there is a better solution I will be glad to change it!

Since we are talking about it, we may think about slightly changing the env variable $ECE3_POSTPROC_ISAC_STRUCTURE to something like $ECE3_POSTPROC_DATA_STRUCTURE: values can be "ISAC" or "BSC" or anything else (and a default will be present in case of variable not declared).

[etiennesky]

Ah ok now I understand better, the eval is for the ifs/nemo output directory, and was needed since you movedthe code to hiresclim_master.sh to avoid code duplication inside the scripts, nice.

I agree on your proposal for ECE3_POSTPROC_DATA_STRUCTURE .

[oloapinivad]

I guess this can be closed, we are happy with the actual flexibility.