search

immerse-project / wp6.2-demonstrator

MIT License
0 stars 0 forks source link

Shared data #26

Open malmans2 opened 2 years ago

malmans2 commented 2 years ago

@atb299

I moved data in #25 to make it easier to reproduce this setup even for people that don't have access to Archer2. All symlinks now point to data in /work/n01/shared/malmans/immerse_wp6.2_data/, with the exception of JRA data that point to /work/n01/shared/nemo/FORCING/JRA/.

If we assume users will download JRA themselves, we could upload /work/n01/shared/malmans/immerse_wp6.2_data/ on Zenodo as it's just 14GB. (The JRA folder below only contains weights)

du -h /work/n01/shared/malmans/immerse_wp6.2_data
41M /work/n01/shared/malmans/immerse_wp6.2_data/1_INPUT
2.3G    /work/n01/shared/malmans/immerse_wp6.2_data/INPUT/JRA
14G /work/n01/shared/malmans/immerse_wp6.2_data/INPUT
892K    /work/n01/shared/malmans/immerse_wp6.2_data/2_INPUT
14G /work/n01/shared/malmans/immerse_wp6.2_data

For non-Archer2 users, the workflow would be this:

git clone git@github.com:immerse-project/wp6.2-demonstrator.git
cd wp6.2-demonstrator
# Download our data from zenodo
cd data/INPUT/JRA
# Download JRA data
malmans2 commented 2 years ago

Other than that, I think users "only" have to add the arch files in wp6.2-demonstrator/arch and edit the compilation and submission scripts in wp6.2-demonstrator/scripts

malmans2 commented 2 years ago

Although non-Archer2 users probably won't use this repo/scripts, but just the nemo-simsar we'll eventually create.

atb299 commented 2 years ago

@malmans2 , the other option for the /shared/ data would be to do something similar to what @jdha has done with e.g. the ORCHESTRA input files. It's less permanent than Zenodo, but still publicly accessible.

malmans2 commented 2 years ago

@jdha has done with e.g. the ORCHESTRA input files

Do you mean make it public on Jasmin? I think it makes sense if we have to share this configuration before it's definitive.

When we are done and we want to publish/make a report, we should probably go for zenodo, and we should add the files that are currently generated using the tools.

atb299 commented 2 years ago

Do you mean make it public on Jasmin? I think it makes sense if we have to share this configuration before it's definitive.

Yes. I'd probably go for the Zenodo option only once we're satisfied that we've got a simulation we want to use for the report/paper.

jdha commented 2 years ago

sounds like a plan - once everything is locked down - then look to publish the forcing etc

jdha commented 2 years ago

I think in one of my setups (possibly a branch of SE-NEMO) if have a ARCHER2 flag - to link to local data - if not grab from remote source