Investigate the use --cpu-count flag with `Models` and `Models --apply`

[DininduSenanayake]

Use the same sample set and Slurm script from #24 to investigate the use of in Models flag -cpu-count

• Below are the efficiency stats we got from the runs without the flag

❯ seff 985625
Job ID: 985625
Cluster: eri
User/Group: dinindu/dinindu
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 4
CPU Utilized: 00:21:03
CPU Efficiency: 77.01% of 00:27:20 core-walltime
Job Wall-clock time: 00:06:50
Memory Utilized: 1.75 GB
Memory Efficiency: 9.71% of 18.00 GB

[DininduSenanayake]

• Submission script

#!/bin/bash -e

#SBATCH --job-name=apsim_models
#SBATCH --output=slurmlogs/%j.out
#SBATCH --cpus-per-task=16
#SBATCH --mem=16G
#SBATCH --time=00:20:00

module load Apptainer
export APPTAINER_BIND="/agr/scratch,/agr/persist"

export APPTAINER_CMD="apptainer exec /agr/persist/projects/2024_apsim_improvements/apsim-simulations/container/apsim-2024.09.7579.0.aimg"

#create .apsimx files : serially
${APPTAINER_CMD} Models --cpu-count ${SLURM_CPUS_PER_TASK} --apply Ahuriri_7a1_13223ConfigFile.txt
${APPTAINER_CMD} Models --cpu-count ${SLURM_CPUS_PER_TASK} --apply Ahuriri_7a1_19487ConfigFile.txt
${APPTAINER_CMD} Models --cpu-count ${SLURM_CPUS_PER_TASK} --apply Ailsa_5a1_13223ConfigFile.txt

#populate .db files : serially
${APPTAINER_CMD} Models --cpu-count ${SLURM_CPUS_PER_TASK} 13223_Ahuriri_7a1.apsimx
${APPTAINER_CMD} Models --cpu-count ${SLURM_CPUS_PER_TASK} 13223_Ailsa_5a1.apsimx
${APPTAINER_CMD} Models --cpu-count ${SLURM_CPUS_PER_TASK} 19487_Ahuriri_7a1.apsimx

Looks quite promising as it gives a slight boost to runtime

❯ seff 985668
Job ID: 985668
Cluster: eri
User/Group: dinindu/dinindu
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 16
CPU Utilized: 00:21:33
CPU Efficiency: 47.54% of 00:45:20 core-walltime
Job Wall-clock time: 00:02:50
Memory Utilized: 3.11 GB
Memory Efficiency: 19.44% of 16.00 GB

• and didn't compromise the integrity of .db files

❯ find ./ -name "*.db" -type f -exec du -h {} + | sort -rh
27M ./19487_Ahuriri_7a1.db
27M ./13223_Ahuriri_7a1.db
26M ./13223_Ailsa_5a1.db

[DininduSenanayake]

Deploy Models and Models --apply to all files in the working directory via for loop. ( Incorporate the --cpu-count from https://github.com/DininduSenanayake/APSIM-eri-mahuika/issues/25)

#!/bin/bash -e

#SBATCH --job-name=apsim_models
#SBATCH --output=slurmlogs/%j.out
#SBATCH --cpus-per-task=16
#SBATCH --mem=16G
#SBATCH --time=00:45:00

module load Apptainer
export APPTAINER_BIND="/agr/scratch,/agr/persist"
export APPTAINER_CMD="apptainer exec /agr/persist/projects/2024_apsim_improvements/apsim-simulations/container/apsim-2024.09.7579.0.aimg"

# Run command for all .txt files, excluding ExampleConfig.txt
for file in *.txt; do
    if [ -f "$file" ] && [ "$file" != "ExampleConfig.txt" ]; then
        ${APPTAINER_CMD} Models --cpu-count ${SLURM_CPUS_PER_TASK} --apply "$file"
    fi
done

# Run command for .apsimx files, excluding specified ones
for file in *.apsimx; do
    if [ -f "$file" ] && [ "$file" != "LargerExample.apsimx" ] && [ "$file" != "2023-10-09_MasterSoilApsimLibrary.apsimx" ]; then
        ${APPTAINER_CMD} Models --cpu-count ${SLURM_CPUS_PER_TASK} "$file"
    fi
done

Job ID is 985810

[DininduSenanayake]

Further investigated and profiled in https://github.com/DininduSenanayake/APSIM-eri-mahuika/issues/26