encapsulate LUMI-related stuff in a separate script

hplt-project / warc2text-runner

Scripts for parallelized extraction of plain texts from WARC archieves. Aiming at common and reproducible extraction approach.

3 stars 0 forks source link

encapsulate LUMI-related stuff in a separate script #13

Open nvanva opened 6 months ago

nvanva commented 6 months ago

move LUMI-related code (srun ...) to a separate script (e.g. stage2job_lumi.sh), provided as an argument to the main starting script; this way we can run the same starting script in different environments just creating a few stage2job* scripts (e.g. lumi/PC, with or without hmlt.zst removal / upload to LUMI-O )