Snakefile 1 - First Production level Snakefile for 06-slurm-array/array_create_db_files.sl

[DininduSenanayake]

Proposed Snakefile

1. Following Snakefile defines a configuration with the Apptainer settings and excluded files.
2. It gets a list of .apsimx files, excluding the specified files.
3. The all rule defines the final output files we want to create.
4. The process_apsimx rule defines how to process each .apsimx file:
5. It requires both the .apsimx file and the corresponding .db ( this is the placeholder .db) file as input.
6. It creates a .processed file as output to mark completion.
   • Perhaps we can modify this later to save file quota
7. sets the resources (memory and time) and number of threads.
8. The shell command loads the Apptainer module, sets the bind path, and runs the Models command.

import os
from glob import glob

# Configuration
config = {
    "apptainer_bind": "/agr/scratch,/agr/persist",
    "apptainer_image": "/agr/persist/projects/2024_apsim_improvements/apsim-simulations/container/apsim-2024.09.7579.0.aimg",
    "excluded_files": ["2023-10-09_MasterSoilApsimLibrary.apsimx", "LargerExample.apsimx"]
}

# Get list of .apsimx files
apsimx_files = [f for f in glob("*.apsimx") if f not in config["excluded_files"]]

rule all:
    input:
        expand("{file}.processed", file=[os.path.splitext(f)[0] for f in apsimx_files])

rule process_apsimx:
    input:
        apsimx = "{file}.apsimx",
        db = "{file}.db"
    output:
        "{file}.processed"
    resources:
        mem_mb = 8000,
        time = "00:10:00"
    threads: 12
    shell:
        """
        module load Apptainer
        export APPTAINER_BIND="{config[apptainer_bind]}"
        apptainer exec {config[apptainer_image]} Models {input.apsimx}
        touch {output}
        """

Configure Snakemake - These instructions are for snakemake > 8

1. Create a configuration directory for Snakemake profiles:

profile_dir="${HOME}/.config/snakemake"
mkdir -p "$profile_dir"

2. Use cookiecutter to create the Slurm profile template:

• If cookiecutter isn't installed, load snakemake module and run pip install --user cookiecutter

template="gh:Snakemake-Profiles/slurm"
cookiecutter --output-dir "$profile_dir" "$template"

3. During the cookiecutter process, you'll be prompted to set values for your profile. For example:

profile_name [slurm]: slurm.my_account
sbatch_defaults []: account=my_account no-requeue exclusive
cluster_sidecar_help: [Use cluster sidecar. NB! Requires snakemake >= 7.0! Enter to continue...]
Select cluster_sidecar:
1 - yes
2 - no
Choose from 1, 2 :
cluster_name []:

4. After completing the prompts, the profile scripts and configuration file will be installed in the $profile_dir as profile_name/[2].

5. You can then use this profile when running Snakemake by adding the --profile flag:

snakemake --profile slurm.my_account ...

This will configure Snakemake to submit jobs to Slurm using the settings you specified in the profile[2].

Remember that the profile typically includes scripts for job submission, status checking, and a configuration file (config.yaml) that defines default values for Snakemake command line arguments. You may need to adjust these files to match your specific Slurm environment and requirements.

[DininduSenanayake]

I have launched this as snakemake --profile slurm --jobs 100 on a tmux session

All of the jobs were submitted under process_apsimx+ job name

1. Expected number of files to process

   ❯ ls *.apsimx | wc -l 
   2614

   • minus the 2 x .apsimx files with soilnames, total number of .apsimx files is = 2612

2. Following output suggests all of the files were processed correctly based on the Slurm state "COMPLTETED"

   ❯ sacct  | grep "process_apsimx+" | grep "COMPLE" | wc -l
   2612

Verify the .db files by using 07-db-file-sorter/db-file-sort.py

• All PASSED ✅

❯ ls PASSED/*.db | wc -l
2612

[DininduSenanayake]

Working Snakefile for this rule : https://github.com/nesi/APSIM-HPC/pull/54