Closed valeriupredoi closed 1 year ago
@sloosvel I am in dire pain after realizing blithering DKRZ's SLURM emails me for every recipe :face_with_spiral_eyes:
@sloosvel what's these jobs up to?
(tool270Test) squeue -u b382109
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
2378977 compute recipe_z b382109 PD 0:00 1 (AssocMaxJobsLimit)
2378976 compute recipe_w b382109 PD 0:00 1 (AssocMaxJobsLimit)
2378975 compute recipe_w b382109 PD 0:00 1 (AssocMaxJobsLimit)
2378974 compute recipe_w b382109 PD 0:00 1 (AssocMaxJobsLimit)
@sloosvel I am in dire pain after realizing blithering DKRZ's SLURM emails me for every recipe face_with_spiral_eyes
You can comment that if it's not useful to you, to me it was!
@sloosvel what's these jobs up to?
I think there is a limit in number of jobs an account can run simultaneously in levante. They will be pending until other jobs finish I guess
@sloosvel what's these jobs up to?
On Levante, a user can't have more than 20 Slurm jobs running at a time. As soon as a job is finished, the next one should start
They will be pending until other jobs finish I guess
Cheers! More emails then :man_facepalming: :rofl:
OK guys - first (and only) sbatch
session over on Levante (I have one stray recipe still running, it's a zombie though) and this is how it looks:
122 out of 127 final
0 out of 127 (1 fixed, not PR-ed yet)
2 out of 127 final
3 out of 127 final
Obsolete/resolved issues comment:
The Julia ones are totally my bad - forgot to install Julia after installing esmvaltool, the autoassess ones are either of the old bug that @alistairsellar is fixing now, or they need aux data that is only on JASMIN, the ones of Missing Data are bothering me badly - since I have turned on auto downloads but they are still missing data, what do you guys recommend doing about those? @sloosvel @remi-kazeroni @bouweandela ? I will post detailed postmortems for the ones that have failed for odd reasons below :+1:
0 out of 127 (1 fixed, not yet PR-ed)
1 out of 127
Hi @valeriupredoi, great job with the testing! I forgot to mention but we have a central pool of downloaded data on Levante at /work/bd0854/DATA/ESMValTool2/download/CMIP6
, /work/bd0854/DATA/ESMValTool2/download/cmip5/output1
, and /work/bd0854/DATA/ESMValTool2/download/cmip5/output1
. Maybe you could add those to your path on top of your download directory? This should help solving the time limit issues (lots of fx files searched on ESGF and/or downloaded I guess).
recipe_smpi.yml - too slow Elapsed time : 04:00:19 (Timelimit=04:00:00)
For this one, I would recommend using:
#SBATCH --partition=compute
#SBATCH --time=08:00:00
#SBATCH --constraint=512G
Indeed, cheers @remi-kazeroni - smpi is a memory gobbler - I restarted it on SLURM and promptly got kicked out coz mem limit (this time around I think all data has been downloaded, hence it went to intensive processing). I'll resubmit with mem reqs. What do you recommend about those that really-really are missing data?
recipe_smpi.yml - too slow Elapsed time : 04:00:19 (Timelimit=04:00:00)
For this one, I would recommend using:
#SBATCH --partition=compute #SBATCH --time=08:00:00 #SBATCH --constraint=512G
even with 512G still fails out of MEM :open_mouth:
oh crap, forgot to change the partition :face_in_clouds:
recipe_smpi.yml - too slow Elapsed time : 04:00:19 (Timelimit=04:00:00)
For this one, I would recommend using:
#SBATCH --partition=compute #SBATCH --time=08:00:00 #SBATCH --constraint=512G
even with 512G still fails out of MEM 😮
You can try with 1024G then! But that's the highest available
recipe_smpi.yml - too slow Elapsed time : 04:00:19 (Timelimit=04:00:00)
For this one, I would recommend using:
#SBATCH --partition=compute #SBATCH --time=08:00:00 #SBATCH --constraint=512G
even with 512G still fails out of MEM open_mouth
You can try with 1024G then! But that's the highest available
totally user-side - forgot to change the partition to compute
- cheers, dude! :beer:
I never managed to run the smpi recipes, @remi-kazeroni did it for me in the last release. Maybe the batch script settings for this recipe can be changed in #2883
with correct SLURM settings as recommended by @remi-kazeroni (:beer:) those smpi monsters are happily plodding along now - yes, we should change the settings for sure. @sloosvel how did you fix the runs for those recipes that really-really dont have data, like I found in https://github.com/ESMValGroup/ESMValTool/issues/2881#issuecomment-1291878142
I don't have a definitive answer for the really-really missing data cases. As said in this comment, you could try to rerun the recipes adding these paths to you config file. But that data pool is 2 releases old. One could argue that we should delete it and re-download everything as /work/bd0854/DATA/ESMValTool2/download/
may contain data retracted from ESGF...
Taking a closer look at some of these (currently) 13 cases:
cmip5/output2
data. You could retry with /work/bd0854/DATA/ESMValTool2/download/cmip5/output2
I think for recipe_climate_change_hotspot.ym, I ended up running it on jasmin
Hi @remi-kazeroni @sloosvel awesome, thanks a lot! Here's the thing(s):
I'll have a closer look at the meeh and schnlund ones, and will ping @schlunma asap
Yes, the version of recipe_flato13ipcc.yml currently in #2156 is running. The cost is to remove/comment out data sets, which do not work on Levante (and to fix a wrong time period for one model). There was already some discussion on how to deal with such cases, and if I remember right @axel-lauer , who is maintainer of the original recipe_flato13ipcc.yml did not agree on removing data sets? It should also be noted, that the option --skip_nonexistent does not work for all diagnostics in recipe_flato13ipcc.yml, because in several data sets from e.g. two different experiments are needed and it does not work, if only one is there. Therefore I was going to ask, which version of recipe_flato13ipcc.yml should be in the end in #2156 in this issue. (Unfortunately I'm also not completely ready with some issues in recipe_flato13ipcc_figures_938_941.yml I hope to finish them soon).
V, can adapt the permission to /scratch/b/b382109/esmvaltool_output
so I can have a look at the logs?
/scratch/b/b382109/esmvaltool_output
@schlunma Manu, they are here /home/b/b382109/manu_logs
Yes, the version of recipe_flato13ipcc.yml currently in #2156 is running. The cost is to remove/comment out data sets, which do not work on Levante (and to fix a wrong time period for one model). There was already some discussion on how to deal with such cases, and if I remember right @axel-lauer , who is maintainer of the original recipe_flato13ipcc.yml did not agree on removing data sets? It should also be noted, that the option --skip_nonexistent does not work for all diagnostics in recipe_flato13ipcc.yml, because in several data sets from e.g. two different experiments are needed and it does not work, if only one is there. Therefore I was going to ask, which version of recipe_flato13ipcc.yml should be in the end in #2156 in this issue. (Unfortunately I'm also not completely ready with some issues in recipe_flato13ipcc_figures_938_941.yml I hope to finish them soon).
@katjaweigel many thanks for your clarification! I will consider this recipe at-risk for now, and will not faff about it until you guys fix it - not the first and not the last time we include not really fully working recipes in a release :grin:
cd: permission denied: /home/b/b382109/manu_logs
:cry:
cd: permission denied: /home/b/b382109/manu_logs
cry
bugger! :face_exhaling: Here they are, bud
I just manually searched for the files on DKRZ's ESGF node and found all files. Not sure what's going on there, but as @remi-kazeroni I would recommend adding our shared pool (/work/bd0854/DATA/ESMValTool2/download/CMIP6
) to your config-user.yml
file :+1:
* recipe_anav13jclim.yml - this is not optimal if "special" cmip5 data is needed, that is not available on ESGF - I would add this recipe to the list of those we have to see what to do about it wrt obsolete data
It's not "special" CMIP5 data, it's rather that our DRS do not take output
into account. See discussion here: https://github.com/ESMValGroup/ESMValTool/issues/2408#issuecomment-1049955903
I just manually searched for the files on DKRZ's ESGF node and found all files. Not sure what's going on there, but as @remi-kazeroni I would recommend adding our shared pool (
/work/bd0854/DATA/ESMValTool2/download/CMIP6
) to yourconfig-user.yml
file +1
cheers, bud! Added, firing those up now :rocket:
OK I added the extra paths and some of them recipes have started plodding along, still - a few doggedly refuse to run still complaining of missing data:
- recipe_anav13jclim.yml
- recipe_check_obs.yml
- recipe_climate_change_hotspot.yml
- recipe_climwip_brunner2019_med.yml
- recipe_collins13ipcc.yml
- recipe_seaice.yml
I'll have to see about running those on JASMIN
recipe_anav13jclim.yml
needs
CMIP5: /work/bd0854/DATA/ESMValTool2/download/cmip5
in addition to the default paths.
FFS man - what's this - we're gathering data like they're sheep on a field in Wales? What are we gonna do about this - suboptimal data storage to put it politely :angry:
I'm rerunning all recipes listed as "Recipes that failed of Missing Data" in this comment and the 2 recipe_bock20jgrfig* listed in the section below but using previously downloaded data in /work/bd0854/DATA/ESMValTool2/download/
. Recipe runs can be found in /scratch/b/b309192/esmvaltool_output
. Current status:
@valeriupredoi, feel free to grab the successful runs and put them in your directory.
EDIT: 4 more successes
Anav dies yet again even with that extra data source :man_facepalming: - JASMIN it is for these showstoppers, only JASMIN is slow like a snail :snail:
This is just our default download directory on Levante for data that has not been provided by DKRZ directly, Remi mentioned that here.
And as mentioned in my previous comment, anav13
is a special case since data from output2
cannot be read with the default DRS (our fault, not CMIPs!). You could also try:
CMIP5: /work/bd0854/DATA/ESMValTool2/download/cmip5/output2
@remi-kazeroni that's brilliant! How you managed to get seaice to run is a true mystery, mine failed like 4 times in the past hour :laughing: - what's your path, bud?
FFS man - what's this - we're gathering data like they're sheep on a field in Wales? What are we gonna do about this - suboptimal data storage to put it politely 😠
Yeah, I know this is not optimal. But this is the directory in which several developers working on Levante download automatically their data to avoid having too many copies of the same datasets... Maybe we should revisit that for the next release and not use previously downloaded data.
@remi-kazeroni that's brilliant! How you managed to get seaice to run is a true mystery, mine failed like 4 times in the past hour 😆 - what's your path, bud?
For the data:
rootpath:
CMIP6: [/work/ik1017/CMIP6/data/CMIP6, /work/bd0854/DATA/ESMValTool2/download/CMIP6]
CMIP5: [/work/kd0956/CMIP5/data/cmip5/output1/, /work/bd0854/DATA/ESMValTool2/download/cmip5/output1, /work/bd0854/DATA/ESMValTool2/download/cmip5/output2]
For the runs: /scratch/b/b309192/esmvaltool_output
. So you have the seaice in: /scratch/b/b309192/esmvaltool_output/recipe_seaice_20221026_142333
This is just our default download directory on Levante for data that has not been provided by DKRZ directly, Remi mentioned that here.
And as mentioned in my previous comment,
anav13
is a special case since data fromoutput2
cannot be read with the default DRS (our fault, not CMIPs!). You could also try:CMIP5: /work/bd0854/DATA/ESMValTool2/download/cmip5/output2
We need to get DKRZ on board to organize/populate their data in their ESGF node - this is a hot mess as it is right now - you guys and me are scraping for data like mad. JASMIN has it much better organized, only problem is JASMIN is abysmally slow compared to Levante and lacking memory. If Levante and Jasmin made a baby, then that'd be perfect :grin:
very many thanks, chaps! I'll let those run (both on me and Remi's partitions) and am off home, tomorrow I'll pick up the results, with all these "missing data" in (or most of them) I'll be able to run the comparison tomorrow - we're on track still :train2:
@schlunma meeh ran fine with the extra Welsh sheep data, bud! Mee-ha! Off to make dinner 🍕
OK guys final count 122/127 recipes successfully run - we are legend :beer: Now, on to the comparison dread :grin:
@sloosvel after trying to make the comparison script work for a bit of a while (not the most straightforwardly code and with quite a few missed catches, @bouweandela - sorry) I have realized that your runs in /scratch/b/b381943/esmvaltool_output
are all empty shells - dir structure is there, but no files at all - can you please tell me what's going on? ASAP, please
@sloosvel also you ran a whole lot of other recipes on top of the standard ESMValTool release ones there - is there anywhere else where you moved the 2.6.0 release output? (one of the things that tripped the compare script)
The outputs you need are in the esmvaltool VM: https://esmvaltool.dkrz.de/shared/esmvaltool/v2.6.0rc4/ . I ran the comparison tool in there because it's where outputs from other versions are. The other outputs are personal work, no need to compare them!
OK cool! How do I get access to those files via a terminal please - I need to run the compare tool via command line, or is there any other way to do that? :beer:
You can log in the machine using your levante credentials: ssh youraccount@esmvaltool.dkrz.de
First move your outputs in /shared/esmvaltool/
(I used rsync excluding the preproc folder for all outputs) and then run the comparison tool against the output for other versions
Thanks - I will do that now. But I am confused by the lack of standardization and disk backuping - I reckon this shouldn't be done for the next release, and the RM should keep the data on the actual Levante disk too eg I am looking at the output from https://esmvaltool.dkrz.de/shared/esmvaltool/v2.6.0/debug.html and see files listed eg /scratch/b/b381943/esmvaltool_output/recipe_autoassess_landsurface_permafrost_20220712_101605/preproc/aa_landsurf_permafrost/tas/CMIP6_ACCESS-CM2_Amon_historical_r1i1p1f1_tas_gn_1992-2002.nc
- those don't exist, what happened to them? Are they behind a virtual OS layer?
OK this is suboptimal to the very least - I managed to get into the VM but it's barren - I need to create a conda env, and depending on the deps (hopefully not many have changed since two days ago when I created the testing env), we may get different results based on what deps the compare script ingests - I may have to use a condalock for that from the actual Levante env, let alone very heavy data duplication (those outputs even without the preproc/ dirs, which I didn't output anyway, are not small). Why didn't you keep the output on Levante, or run there in the first place?
Because I don't have the output for other versions in Levante. Other releases were ran in Mistral. The outputs are in the virtual machine. And it's indicated in the documentation anyway: https://docs.esmvaltool.org/en/latest/utils.html#comparing-recipe-runs
and on top of this all I don't have write permissions to /shared/esmvaltool
to move the data - I really don't want to move it in my $HOME on the VM first and then move them back or symlink them; @remi-kazeroni who is supposed to give me rwx+ to that partition please?
Sister and logical evolution of #2852 - I am commencing testing and comparison of recipes and recipes results in order to release 2.7.0 at the end of this week (hopefully). System parameters below, work done on DKRZ/Levante: submit files in
/home/b/b382109/submit
, output in/scratch/b/b382109/esmvaltool_output
System and settings
conda
/mamba
Git branch and state
Date: 25 October 2022 14:22 BST
Environment
On Levante:
Environment file
ToolEnv270Test.yml
Extraneous file movements
I moved the autoassess-specific files to
/home/b/b382109/autoassess_files
- run was succesful for AA recipes then :+1:Ad-hoc hacks (code changes)
/home/b/b382109/ESMValTool/esmvaltool/diag_scripts/land_carbon_cycle/diag_global_turnover.py
l.278 change.outline_patch
with.spines["geo"]
as suggested by @zklaus in https://github.com/ESMValGroup/ESMValTool/issues/2886#issuecomment-1292135500 (cheers, dude!) - this will have to be PR-edMods to config user file
Added DKRZ downloaded data pool as:
as @schlunma and @remi-kazeroni have suggested :beer:
Recipe runs
Recipe runs results (as of final on 27 October 2022) are listed in https://github.com/ESMValGroup/ESMValTool/issues/2881#issuecomment-1291878142 (with very many thanks to @remi-kazeroni for running the impossible to run ones!) and are as follows:
(*)
means not counting/counting the one that had a DiagnosticError but was fixed but not PR-edRunning the comparison
Login and access to the DKRZ esmvaltool VM
Results from recipe runs are stored on the VM; login with:
Get and install miniconda on VM
E.g.
scp Miniconda3-py39_4.12.0-Linux-x86_64.sh b382109@esmvaltool.dkrz.de:~
from a file already on Levante.Setting up the input files
If you wrote recipe runs output to Levante
/scratch
partition be aware that the data will be removed after two weeks, so you will have to move the output data to the/work
partition, via e.g. anohup
job:/work
is visible by the VM so you can run the compare tool straight on the VM.NOTE do not store final release results on the VM including
/preproc/
dirs, the total size for all the recipes output, including/preproc/
dirs is in the 4.5TB ballpark, much too high for the VM storage capacityRunning compare tool at VM
tool270Compare
release270stable
pip install imagehash
Input/output/run
/work/bd0854/b382109/v270
(containspreproc/
dirs too, 122 recipes)/mnt/esmvaltool_disk2/shared/esmvaltool/v2.6.0rc4
(does not containpreproc/
dirs)nohup python ESMValTool/esmvaltool/utils/testing/regression/compare.py /mnt/esmvaltool_disk2/shared/esmvaltool/v2.6.0rc4 /work/bd0854/b382109/v270 > compare270output.txt
Sanity check, as outputted by
compare.py
First pass result
Running the
compare.py
results in a few recipes not-OK (NOK) wrt plots differing from previous release v2.6.0, summary in https://github.com/ESMValGroup/ESMValTool/issues/2881#issuecomment-1294735465Detailed plots inspection
Plots that differ for the 34 recipes that have them different is happening in https://github.com/ESMValGroup/ESMValTool/issues/2881#issuecomment-1295001054