search

plesager / ece3-postproc

Suite of processing tools for EC-Earth3 output
5 stars 8 forks source link

mods for autosubmit (see issue #9) #11

Closed etiennesky closed 6 years ago

etiennesky commented 6 years ago

Hi I have implemented a first version of flexible rundir, postdir and diagdir. Should work as previously but some env. vars have to be changed.

etiennesky commented 6 years ago

oh and this now works for me in the autosubmit workflow, by adding these vars to the calling script

ROOTDIR=%CURRENT_ROOTDIR%
START_date=%SDATE%
MEMBER=%MEMBER%
RUN_dir=$ROOTDIR/$START_date/$MEMBER/runtime

    # variables which do not depend on experiment
    export ECE3_POSTPROC_TOPDIR=$HOME/ecearth3/ece3-postproc #<dir where this file is>
    export ECE3_POSTPROC_DATADIR=/gpfs/projects/bsc32/repository/ece3-postproc # <dir where your ecearth data are located -see your config-run.xml>  
    export ECE3_POSTPROC_MACHINE=mn4 #<name of your (HPC) machine>
    export ECE3_POSTPROC_ACCOUNT=bsc32 #<HPC account>

    # variables which are specific to experiment/start date/member
    export ECE3_POSTPROC_RUNDIR='$RUN_dir'
    export ECE3_POSTPROC_POSTDIR='$RUN_dir/ece3post'
    export ECE3_POSTPROC_DIAGDIR='$RUN_dir/ece3diag'
    export RUN_dir

RUN_dir is

plesager commented 6 years ago

I merged your code in 5cf743d, but also tried to tidy and clean things up. I removed some of the requirements on the env variables. Some variables have been moved to their respective config file. I wrote the eval_dirs function (in functions.sh) which acts as a generic parser, still based on eval, and is applied a lot throughout hiresclim2. I think this is quite flexible now, and any kind of model output tree can be specified.

@oloapinivad and @abalanzino:

Finally the non-yearly legs is flexible enough that it works for 1, 2, 3, 4, 6 and 12-month legs (and maybe mulitples of 12?). Still experimental, and I have no clue if that works with NEMO, so I put a stop there.

oloapinivad commented 6 years ago

Thanks @plesager, I merged your changes in our code. I am testing it right now: in the last version I have seen that we partially lost the support for analyzing experiments from different users since the templates wasn't operational when ALTDIR is used.

I have done a test reintroduced it using a $COREIFSDIR and $CORENEMODIR (see _conf_hiresclimmarconi.sh in https://github.com/oloapinivad/ece3-postproc/commit/26ef233ff40f8a3b2227aa5951cc213368232213) that is working. However, I am not convinced this is the best approach. A flag -u USER that is evaluated by _masterhirescim.sh may be cleaner: this will use a default $USER or a specific one. What do you think? I can work on it, just want to hear your opinion on this.

Also, If we evaluate the full path (including the user), I also feel that the ECE3_POSTPROC_RUNDIR will become deprecated.

plesager commented 6 years ago

Yes the ECE3_POSTPROC_RUNDIR is obsolete now (I have updated the README in that regard). I have only propagate the changes to hiresclim2 and ecmean, but not to awmg or timeseries (which I have not tested in ages), so you will still find reference for it there.

I understand what you are trying to do with ALTDIR. This is not easy, since it does not define the full tree, and even if we decided to fully evaluate ALTDIR (which can be done) we would still have to find out what are the full path to IFS and NEMO results. Now it only assumes a default for those. Your idea of a USER flag is the same as using another env variable like USERexp you had before. Which works fine if you can just change the USER (yours) with USERexp and everything else is the same in the template. Then IFSRESULTS0 and NEMORESULTS0 should suffice, by exporting ${USERexp:=$USER} at the top of _confhiresclim[platform].sh_, and defining USERexp when needed. But to easily switch at the command line, this has to be combined with a USER flag as you propose. Is it your scenario: having an easy switch of user at the command line that could apply to IFSRESULTS0 and NEMORESULTS0? I woudln't mind having such a USER switch at the CLI, and I think we could retire ALTDIR altogether.

oloapinivad commented 6 years ago

Hi,

Yes this is exactly my point. Often we have users that run the analysis that are not the owner of the experiments. So we would like to change the IFSRESULTS0 and NEMORESULTS0 but only at user level. Probably it is enough to do something like -> if a -u USERexp is specified, apply a sed on the $USER inside the IFSRESULTS0 and NEMORESULTS0 string replacing with the USERexp, without the need of defining an env variable USERexp. In theory, this can work in parallel with ALTDIR, but I am agree that ALTDIR is not very useful in the very end. If you agree, I will implement this in the next days.

2018-02-28 11:23 GMT+01:00 P. Le Sager notifications@github.com:

Yes the ECE3_POSTPROC_RUNDIR is obsolete now (I have updated the README in that regard). I have only propagate the changes to hiresclim2 and ecmean, but not to awmg or timeseries (which I have not tested in ages), so you will still find reference for it there.

I understand what you are trying to do with ALTDIR. This is not easy, since it does not define the full tree, and even if we decided to fully evaluate ALTDIR (which can be done) we would still have to find out what are the full path to IFS and NEMO results. Now it only assumes a default for those. Your idea of a USER flag is the same as using another env variable like USERexp you had before. Which works fine if you can just change the USER (yours) with USERexp and everything else is the same in the template. Then IFSRESULTS0 and NEMORESULTS0 should suffice, by exporting ${USERexp:=$USER} at the top of conf_hiresclim[platform].sh_, and defining USERexp when needed. But to easily switch at the command line, this has to be combined with a USER flag as you propose. Is it your scenario: having an easy switch of user at the command line that could apply to IFSRESULTS0 and NEMORESULTS0? I woudln't mind having such a USER switch at the CLI, and I think we could retire ALTDIR altogether.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/plesager/ece3-postproc/pull/11#issuecomment-369194830, or mute the thread https://github.com/notifications/unsubscribe-auth/AbCe6929GfVZvaUSzAnQkYIc-mpqNykTks5tZSk0gaJpZM4SRMfM .

etiennesky commented 6 years ago

maybe we should move this discussion to an issue? I just created another pull request with minor changes and a suggestion