So this doesn't fall below the radar, I'm just posting it here.
In summary, I'm trying to start a new suite, but one which includes runs that have already been analysed by others. Unusually, I'm working from a clean install of BGCVal2, so there should be no issues with me using old scripts. The output below is what's going wrong:
(bgcval2) [ayool@sci2 bgcval2]$ analysis_compare -y input_yml/new_proto_ukesm2.yml
analysis_timeseries: Comparison config file input_yml/new_proto_ukesm2.yml
---------------------
timeseries_compare: new_proto_ukesm2
job ids: dict_keys(['u-by230', 'u-ct200', 'u-ct739'])
u-by230 description: UKESM1.1 piControl
u-by230 colour: black
u-by230 line thickness & style: 0.7 -
u-by230 Shift time by 0.0
u-by230 Time range (None means all): None
u-by230 suite: kmf_phys physics_no_mld
u-by230 auto_download True
u-ct200 description: GC5
u-ct200 colour: red
u-ct200 line thickness & style: 2.0 -
u-ct200 Shift time by 0.0
u-ct200 Time range (None means all): None
u-ct200 suite: kmf_phys physics_no_mld
u-ct200 auto_download True
u-ct739 description: GC5-ESM-proto
u-ct739 colour: blue
u-ct739 line thickness & style: 2.0 -
u-ct739 Shift time by 0.0
u-ct739 Time range (None means all): None
u-ct739 suite: kmf_phys physics_no_mld
u-ct739 auto_download True
Will use parameters from /home/users/ayool/bgcval2/bgcval2/default-bgcval2-config.yml for runtime.
Configuration: WARNING [standard-paths][general] Not checking machinelocation: JASMIN
Configuration: [standard-paths][general] machinelocation: JASMIN
Configuration: [standard-paths][general] root_dir: /gws/nopw/j04/ukesm
Configuration: WARNING [standard-paths][general] Not checking shelvedir: /gws/nopw/j04/ukesm/BGC_data/ayool/bgcval2/shelves
Configuration: [standard-paths][general] shelvedir: /gws/nopw/j04/ukesm/BGC_data/ayool/bgcval2/shelves
Configuration: WARNING [standard-paths][general] Not checking p2p_ppDir: /gws/nopw/j04/ukesm/BGC_data/bgcval2/ukesm_postProcessed
Configuration: [standard-paths][general] p2p_ppDir: /gws/nopw/j04/ukesm/BGC_data/bgcval2/ukesm_postProcessed
Configuration: WARNING [standard-paths][general] Not checking imagedir: /gws/nopw/j04/ukesm/BGC_data/ayool/bgcval2/images
Configuration: [standard-paths][general] imagedir: /gws/nopw/j04/ukesm/BGC_data/ayool/bgcval2/images
Configuration: [standard-paths][general] ModelFolder_pref: /gws/nopw/j04/ukesm/BGC_data
Configuration: [standard-paths][general] shared_mass_scripts: /gws/nopw/j04/esmeval/bgcval2/shared_mass_scripts/
Configuration: [standard-paths][general] orcaGridfn: /gws/nopw/j04/esmeval/bgc-val/mesh_mask_eORCA1_wrk.nc
Configuration: [standard-paths][general] orca1bathy: /gws/nopw/j04/esmeval/bgc-val/ORCA1bathy.nc
Configuration: [standard-paths][general] ObsFolder: /gws/nopw/j04/esmeval/example_data/bgc
Configuration: [standard-paths][data-files] Dustdir: /gws/nopw/j04/esmeval/example_data/bgc/MahowaldDust/
Configuration: [standard-paths][data-files] WOAFolder_annual: /gws/nopw/j04/esmeval/example_data/bgc/WOA/annual/
Configuration: [standard-paths][data-files] WOAFolder: /gws/nopw/j04/esmeval/example_data/bgc/WOA/
Configuration: [standard-paths][data-files] DMSDir: /gws/nopw/j04/esmeval/example_data/bgc/DMS_Lana2011nc/
Configuration: [standard-paths][data-files] MAREDATFolder: /gws/nopw/j04/esmeval/example_data/bgc/MAREDAT/MAREDAT/
Configuration: [standard-paths][data-files] GEOTRACESFolder: /gws/nopw/j04/esmeval/example_data/bgc/GEOTRACES/GEOTRACES_PostProccessed/
Configuration: [standard-paths][data-files] GODASFolder: /gws/nopw/j04/esmeval/example_data/bgc/GODAS/
Configuration: [standard-paths][data-files] TakahashiFolder: /gws/nopw/j04/esmeval/example_data/bgc/Takahashi2009_pCO2/
Configuration: [standard-paths][data-files] MLDFolder: /gws/nopw/j04/esmeval/example_data/bgc/IFREMER-MLD/
Configuration: [standard-paths][data-files] iMarNetFolder: /gws/nopw/j04/esmeval/example_data/bgc/LestersReportData/
Configuration: [standard-paths][data-files] GlodapDir: /gws/nopw/j04/esmeval/example_data/bgc/GLODAP/
Configuration: [standard-paths][data-files] GLODAPv2Dir: /gws/nopw/j04/esmeval/example_data/bgc/GLODAPv2/GLODAPv2_Mapped_Climatologies/
Configuration: [standard-paths][data-files] OSUDir: /gws/nopw/j04/esmeval/example_data/bgc/OSU/
Configuration: [standard-paths][data-files] CCIDir: /gws/nopw/j04/esmeval/example_data/bgc/CCI/
Configuration: [standard-paths][data-files] icFold: /gws/nopw/j04/esmeval/example_data/bgc/InitialConditions/
deleteBadLinksAndZeroSize: find /gws/nopw/j04/ukesm/BGC_data/u-by230//. -size 0 -print -delete
find: cannot delete ‘/gws/nopw/j04/ukesm/BGC_data/u-by230//./1y’: Directory not empty
find: cannot delete ‘/gws/nopw/j04/ukesm/BGC_data/u-by230//.’: Invalid argument
deleteBadLinksAndZeroSize: find -L /gws/nopw/j04/ukesm/BGC_data/u-by230//. -type l -delete -print
deleteBadLinksAndZeroSize: find /gws/nopw/j04/ukesm/BGC_data/u-by230//. -size 0 -print -delete
find: cannot delete ‘/gws/nopw/j04/ukesm/BGC_data/u-by230//./1y’: Directory not empty
find: cannot delete ‘/gws/nopw/j04/ukesm/BGC_data/u-by230//.’: Invalid argument
deleteBadLinksAndZeroSize: find -L /gws/nopw/j04/ukesm/BGC_data/u-by230//. -type l -delete -print
download_from_mass: Downloading at the following files:
download_from_mass: the command is (dry-run):
moo get --fill-gaps moose:/crum/u-by230/ony.nc.file/*.nc /gws/nopw/j04/ukesm/BGC_data/u-by230/
writing file: mass_scripts/u-by230.sh
file contents:
# Run this script on mass-cli1.jasmin.ac.uk
# from login1.jasmin.ac.uk, ssh to the mass machine:
# ssh -X mass-cli
# run script with:
# source /home/users/ayool/bgcval2/mass_scripts/u-by230.sh
# moo passwd -r # if mass password is expired
source /etc/bashrc
moo get --fill-gaps moose:/crum/u-by230/ony.nc.file/*.nc /gws/nopw/j04/ukesm/BGC_data/u-by230/
writing file in shared path /gws/nopw/j04/esmeval/bgcval2/shared_mass_scripts/u-by230.sh
Traceback (most recent call last):
File "/home/users/ayool/miniconda3/envs/bgcval2/bin/analysis_compare", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/users/ayool/bgcval2/bgcval2/analysis_compare.py", line 677, in main
load_yml_and_run(compare_yml, config_user, skip_timeseries)
File "/home/users/ayool/bgcval2/bgcval2/analysis_compare.py", line 589, in load_yml_and_run
download_from_mass(jobID, doMoo=do_mass_download, auto_download=auto_download[jobID], config_user=config_user)
File "/home/users/ayool/bgcval2/bgcval2/download_from_mass.py", line 487, in download_from_mass
shutil.copy(download_script_path, shared_file_path)
File "/home/users/ayool/miniconda3/envs/bgcval2/lib/python3.11/shutil.py", line 419, in copy
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "/home/users/ayool/miniconda3/envs/bgcval2/lib/python3.11/shutil.py", line 258, in copyfile
with open(dst, 'wb') as fdst:
^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/gws/nopw/j04/esmeval/bgcval2/shared_mass_scripts/u-by230.sh'
The problem seems to be with file /gws/nopw/j04/esmeval/bgcval2/shared_mass_scripts/u-by230.sh whose permissions permit reading, but not writing for anyone other than the user that created it. (Curiously, I'm also not sure why it's writing this at all - I've already got my own version of this file in my local directory.)
As an aside, a few observations:
The specific file is readable by me, but not writeable, even for group members like me
In the same directory there appear to be files from earlier this year with filenames that append the username to the job ID
This practice, which presumably would let me write my own version of the offending file, seems to have stopped in mid-April
The bespoke lazy solution might be to give all of the similar scripts for the jobIDs that I'm interested in write permission, that way my analysis should proceed. However, that's not really a good solution. As noted above, giving users their bespoke versions of these scripts - as happened in the past - may be an alternative worth considering. These files are very small, so the overhead is limited. That said, it seems likely that, if I can't write to these files, I might not be able to write to the processed shelve files down the line. My script hasn't gotten far enough for me to see whether this is also the case.
In any case, this problem didn't exist before, so either JASMIN has done something stupid around permissions (not unlikely given their fetish for disk space rearranging), or a new piece of BGCVal2 is a little bit too clever and is being hoisted by issues that don't occur with new jobIDs or where a user re-visits a jobID that they created in the past.
Cheers.
UPDATE: So, I got my suite to run by removing the offending job ID. Not entirely satisfactory, however, since I kind-of needed the job in there.
In passing, before my suite ran, I had to fiddle about again with the mixed layer depth diagnostics. This is set by default as mlotst, but at least one of the runs I was looking at didn't have this or complained. In the end, I tweaked the key_files/mld.yml file to include two alternatives for this, somxl010 and somxzint1. Seems to have kind-of worked. At least, I get something coherent in the suite ...
So this doesn't fall below the radar, I'm just posting it here.
In summary, I'm trying to start a new suite, but one which includes runs that have already been analysed by others. Unusually, I'm working from a clean install of BGCVal2, so there should be no issues with me using old scripts. The output below is what's going wrong:
The problem seems to be with file
/gws/nopw/j04/esmeval/bgcval2/shared_mass_scripts/u-by230.shwhose permissions permit reading, but not writing for anyone other than the user that created it. (Curiously, I'm also not sure why it's writing this at all - I've already got my own version of this file in my local directory.)As an aside, a few observations:
The bespoke lazy solution might be to give all of the similar scripts for the jobIDs that I'm interested in write permission, that way my analysis should proceed. However, that's not really a good solution. As noted above, giving users their bespoke versions of these scripts - as happened in the past - may be an alternative worth considering. These files are very small, so the overhead is limited. That said, it seems likely that, if I can't write to these files, I might not be able to write to the processed shelve files down the line. My script hasn't gotten far enough for me to see whether this is also the case.
In any case, this problem didn't exist before, so either JASMIN has done something stupid around permissions (not unlikely given their fetish for disk space rearranging), or a new piece of BGCVal2 is a little bit too clever and is being hoisted by issues that don't occur with new jobIDs or where a user re-visits a jobID that they created in the past.
Cheers.
UPDATE: So, I got my suite to run by removing the offending job ID. Not entirely satisfactory, however, since I kind-of needed the job in there.
In passing, before my suite ran, I had to fiddle about again with the mixed layer depth diagnostics. This is set by default as
mlotst, but at least one of the runs I was looking at didn't have this or complained. In the end, I tweaked thekey_files/mld.ymlfile to include two alternatives for this,somxl010andsomxzint1. Seems to have kind-of worked. At least, I get something coherent in the suite ...CompareReports/bgcval2/ayool/proto_ukesm2_v2Anyway, I'm flipping to making a new suite for now, but the issues here probably still need looking into.