dcache not reliable enough for education

[BSchilperoort]

Occasionally Surf's dcache in not accessible, for hours at a time. This is annoying for researchers (or during demo's). However, if you use ewatercycle for teaching this is terrible.

@sverhoeven suggested rehosting a subset of what's on dcache on a SRC virtual machine, which can then be mounted on the machines which the students will work on.

[BSchilperoort]

@RolfHut

[RolfHut]

Thank you @BSchilperoort for raising the issue. For education currently we only use the dCache data for generating forcing, so we would also be helped if the import ewatercycle (and mainly: import ewatercycle.forcing) doesn't fail when paths are not available, but only throws an error when paths are actually accessed. This would allow students that only need to run models to still do so, even if dCache is down.

[sverhoeven]

I tried https://servicedesk.surf.nl/wiki/display/WIKI/Create+a+shared+storage+for+Linux on https://grader35.ewatercycle-nle.src.surf-hosted.nl/ with /data/shared being a mounted remote storage.

I was able to run apptainer and plot a netcdf file.

Operational steps:

1. Create file server workspace
2. Populate file server
3. Create teaching machines

Cons:

• extra machine with its own storage item
• file server must be in same collaborative organization as teaching machines
• Samba server catalog item needs slight tweaking, could make new catalog item if needed
  • client for dcache like rclone with creds
  • share should be read only

Pros:

• only need dcache working while populating file server
• smaller then dcache, restricted by storage items offered by Surf Research Cloud
• main branch and grader branch are very different
• no longer need cache storage item on teaching machine

Todos:

• populate file server with training material + matching ewatercycle.yaml
• add https://gitlab.com/rsc-surf-nl/plugins/plugin-samba/-/blob/main/samba-client-linux.yml to our roles
  • modify /etc/fstab to mount read only and use root only readable creds
• make /etc/ewatercycle.yaml symlink to /data/shared/ewatercycle.yaml + replace /mnt/data with /data/shared
• delete dcache mount stuff
• (optional) host training material on non-src resources so material can be downloaded during creation of file server.

Notes

Create storage item called `shared` for ewatercycle-nlesc In ewatercycle-nlesc collab org add secret samba_password: Create private network for ewatercycle-nlesc name: file-storage-network Create samba file server attach: - storage: shared - network: file-storage-network name: fs1 description: File server for eWatercycle teaching machines On server In smb.conf set to read only = yes Restart samba `systemctl restart smbd` Populate ``` cd /data/volume_2/samba-share mkdir singularity-images wget -O singularity-images/ewatercycle-pcrg-grpc4bmi_setters.sif 'https://webdav.grid.surfsara.nl/singularity-images/ewatercycle-pcrg-grpc4bmi_setters.sif?action=show&authz=' mkdir -p climate-data/obs6/Tier3/ERA5 wget -O climate-data/obs6/Tier3/ERA5/OBS6_ERA5_reanaly_1_day_pr_1995-1995.nc 'https://webdav.grid.surfsara.nl/climate-data/obs6/Tier3/ERA5/OBS6_ERA5_reanaly_1_day_pr_1995-1995.nc?action=show&authz=' ``` Create teaching machine - Add private network Mount samba share Install + configure similar to https://gitlab.com/rsc-surf-nl/plugins/plugin-samba/-/blob/main/samba-client-linux.yml ``` nmap -p 445 -T4 -v 10.10.10.0/24 | awk -F'[ /]' '/Discovered open port/{print $NF}' 10.10.10.44 # in /etc/fstab //10.10.10.44/samba-share /data/shared cifs username=smbuser,password=,ro,cache=loose # replace password= with credentials file which is not readable by others mount -a ``` Test ``` from grpc4bmi.bmi_client_apptainer import BmiClientApptainer !mkdir /tmp/work client = BmiClientApptainer('/data/shared/singularity-images/ewatercycle-pcrg-grpc4bmi_setters.sif',work_dir='/tmp/work') client.get_component_name() 'pcrglobwb' del client import xarray as xr ds = xr.open_dataset('/data/shared/climate-data/obs6/Tier3/ERA5/OBS6_ERA5_reanaly_1_day_pr_1995-1995.nc') ds ds['pr'].isel(time=15).plot() ```

[RolfHut]

Thank you @sverhoeven . I think I got most of that :-). Relating to "(optional) host training material on non-src resources so material can be downloaded during creation of file server.". Maybe we can host this on a repo like Zenodo, 4TU? Or if it is purely for teaching we can even make it available as part of Open Educational Resources?

[BSchilperoort]

We could host the data somewhere, but we would have to decide what exactly, and solve some potential issues:

• apptainer images (for running containerized models). *
• parameter sets for models you want to include (have to make sure we don't violate any copyright)
• CMORized ERA5 data (which need a lot of storage... 4TU would be possible, if we can stay under 1TB**)

* Alternative could be a script to generate them from docker images? not sure what's better @sverhoeven

**At TU Delft you're can upload 1TB/year for free. Zenodo is max 50GB.