Run with slurm

[rburghol]

This is very rough draft prototype examples. Todo:

• [ ] Find out how to specify the slurm output directory
• [x] Figure out how to send individual jobs to maximize thread/core use
• [x] Explore P6 basin order type execution: https://github.com/HARPgroup/cbp_wsm/issues/86
• [x] Test dependencies for riversegs (See River slurm Dependency below)
• [x] install slurmdbd (see https://github.com/HARPgroup/vahydro/wiki/Sysadmin_setuplog)
• [ ] #5

Cancelling jobs:

sjobs=`squeue --format="%.4i"`
for i in $sjobs; do scancel $i; done

View slurm Jobs with Full Names

• New command: slurmq, which is an alias for:
  • squeue --format="%.18i %.9P %.36j %.8u %.8T %.10M %.9l %.6D %R"
  • yields:

         JOBID PARTITION                                 NAME     USER    STATE       TIME TIME_LIMI  NODES NODELIST(REASON)
          1953     debug      p6_vadeq_subsheds_JU2_6810_6900      rob  PENDING       0:00 UNLIMITED      1 (Dependency)
          1954     debug      p6_vadeq_subsheds_JU3_6900_6950      rob  PENDING       0:00 UNLIMITED      1 (Dependency)
          1955     debug      p6_vadeq_subsheds_JU3_6950_7330      rob  PENDING       0:00 UNLIMITED      1 (Dependency)
          1956     debug      p6_vadeq_subsheds_JU4_7330_7000      rob  PENDING       0:00 UNLIMITED      1 (Dependency)
          1957     debug      p6_vadeq_subsheds_JU4_7000_7300      rob  PENDING       0:00 UNLIMITED      1 (Dependency)
          1952     debug      p6_vadeq_subsheds_JU2_6600_6810      rob  RUNNING       3:51 UNLIMITED      1 deq2

    Basic Config Needed

    All examples expect that the model is already configured, with basin and scenario

. hspf_config
basin=JA5_7480_0001
segs=`cbp get_riversegs $basin`
scenario="subsheds"
model="hspf_cbp6" # alt hsp2_cbp6, etc.

River slurm Dependency

• Run a basin with slurm dependency
• This considers both land and river segment runs (if submitted with the job name using:

  job_name=`cbp mm_job_name $scenario $i`

• Use a script called mm_river_deps to automatically account for the model directory, scenario, and segment. View code
• Will fail if there are multiple jobs with the same exact name, i.e., if someone has erroneously run to slurm jobs for the same directory/scenario/segment, so this is an edge case, but it can happen.

segs=`cbp get_riversegs $basin`
for i in $segs; do
  deps=`cbp mm_river_deps $scenario $i`
  echo "Segment $i Found deps= $deps"
  job_name=`cbp mm_job_name $scenario $i`
  echo "sbatch --job-name=$job_name $deps /opt/model/meta_model/run_model $model $scenario $i  auto river"
  sbatch --job-name=$job_name $deps /opt/model/meta_model/run_model $model $scenario $i auto river
done

River Upstream & Land Simulation Dependencies

Trigger the job with:

# run land
segs=`cbp get_landsegs $basin`
location="p6_vadeq" # crucial to avoid collisions if running multiple installs
read -r scope land_scenario < <(cbp get_config $scenario river "LAND SCENARIO")
for i in $segs; do
  job_name="${location}_${scenario}_${i}"
  echo "sbatch --job-name=\"$job_name\" /opt/model/meta_model/run_model $model $land_scenario $i auto land "
  sbatch --job-name="$job_name" /opt/model/meta_model/run_model $model $land_scenario $i auto land 
done
# run river
# can now immediately be submitted after submitting land 
# since the dependency ordering works with land and river 
segs=`cbp get_riversegs $basin`
for i in $segs; do
  deps=`cbp mm_river_deps $scenario $i`
  echo "Segment $i Found deps= $deps"
  job_name=`cbp mm_job_name $scenario $i`
  echo "sbatch --job-name=$job_name $deps /opt/model/meta_model/run_model $model $scenario $i  auto river"
  sbatch --job-name=$job_name $deps /opt/model/meta_model/run_model $model $scenario $i auto river
done

Show the Dependencies

• Use a small script with formatting called slurmq, which just does the following: squeue --format="%.18i %.9P %.36j %.8u %.8T %.10M %.9l %.6D %R"

             JOBID PARTITION                                 NAME     USER    STATE       TIME TIME_LIMI  NODES NODELIST(REASON)
              3390     debug      p6_vadeq_subsheds_YM1_6370_6620      rob  PENDING       0:00 UNLIMITED      1 (Dependency)
              3391     debug      p6_vadeq_subsheds_YM2_6122_6120      rob  PENDING       0:00 UNLIMITED      1 (Dependency)
              3392     debug      p6_vadeq_subsheds_YM2_6120_6430      rob  PENDING       0:00 UNLIMITED      1 (Dependency)
              3393     debug      p6_vadeq_subsheds_YM3_6430_6620      rob  PENDING       0:00 UNLIMITED      1 (Dependency)
              3394     debug      p6_vadeq_subsheds_YM4_6620_0001      rob  PENDING       0:00 UNLIMITED      1 (Dependency)
              3385     debug             p6_vadeq_subsheds_N51033      rob  RUNNING       0:43 365-00:00:00      1 deq2
              3386     debug             p6_vadeq_subsheds_N51097      rob  RUNNING       0:43 365-00:00:00      1 deq2
              3387     debug             p6_vadeq_subsheds_N51137      rob  RUNNING       0:43 365-00:00:00      1 deq2
              3388     debug             p6_vadeq_subsheds_N51177      rob  RUNNING       0:43 365-00:00:00      1 deq2
              3389     debug             p6_vadeq_subsheds_N51101      rob  RUNNING       0:43 365-00:00:00      1 deq2

Meteorology Creation and Land Run with slurm dependencies

• This requires extensive variable setup that is not included below, see https://github.com/HARPgroup/model_meteorology/issues/20

  
  for i in $segs; do
  # assemble the met dataset
  met_job=`cbp mm_job_name $met_scen $i`
  cd $NLDAS_ROOT
  sbatch --job-name="$met_job" ./make_met2date $i $mtds $mtde "$refresh_years" landseg
  # run the QA
  cd $MODEL_ROOT
  met_id=`cbp mm_job_id $met_job` # this will be a dependency for several down the line
  qa_job=`cbp mm_job_name qa_$met_scen $i`
  sbatch --job-name=$qa_job --dependency=$met_id $NLDAS_ROOT/sqa_met $i met2date $lseg_ftype $model_version "$refresh_years"
  # now generate the WDM
  wdm_job=`cbp mm_job_name "wdm_${met_scen}" $i`
  sbatch --job-name=$wdm_job --dependency=$met_id wdm_pm_one $i $sdate $edate $model_dataset $model_subset $met_scen $prad_scen $met_dataset overwrite
  wdm_id=`cbp mm_job_id $wdm_job`
  land_job=`cbp mm_job_name $land_scenario $i`
  sbatch --job-name=$land_job --dependency=$wdm_id /opt/model/meta_model/run_model hspf_cbp5 $land_scenario $i auto land
  done

### Behaviors
- to slurm or not to slurm:
  - NO: within `run_model` if args are included, use slurm otherwise just execute script?
  - YES: or standalone controller that calls `run_model`? 
    - best bet is standalone, since `run_model` is really the method to run a single, self-contained segment
  - see here for coding bash args https://www.redhat.com/sysadmin/arguments-options-bash-scripts
  - each object class, i.e. land, and river, must have a `get_dependencies()` function specific to the model domain to enable optimal SLURM Dependencies
  - Cache dates can be dependent on file last modified?

#### Land Use generation
- Test 1: treat each land segment as a batch
  - thus they can occupy different cores
  - and they don't depend on each other

segs=cbp get_landsegs P for i in $segs; do echo "sbatch /opt/model/meta_model/run_model hspf_cbp6 vadeq_2021 $i auto land prep" sbatch /opt/model/meta_model/run_model hspf_cbp6 vadeq_2021 $i auto land prep done

- This queues them up, with 7 processes running simultaneously (not counting the 1 process from another model)

squeue squeue: error: NodeNames=deq2 Sockets=0 is invalid, reset to 1 JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 361 debug run_mode rob PD 0:00 1 (Resources) 362 debug run_mode rob PD 0:00 1 (Priority) 363 debug run_mode rob PD 0:00 1 (Priority) 364 debug run_mode rob PD 0:00 1 (Priority) 365 debug run_mode rob PD 0:00 1 (Priority) 366 debug run_mode rob PD 0:00 1 (Priority) 367 debug run_mode rob PD 0:00 1 (Priority) 368 debug run_mode rob PD 0:00 1 (Priority) 369 debug run_mode rob PD 0:00 1 (Priority) 370 debug run_mode rob PD 0:00 1 (Priority) 371 debug run_mode rob PD 0:00 1 (Priority) 354 debug run_mode rob R 0:00 1 deq2 355 debug run_mode rob R 0:00 1 deq2 356 debug run_mode rob R 0:00 1 deq2 357 debug run_mode rob R 0:00 1 deq2 358 debug run_mode rob R 0:00 1 deq2 359 debug run_mode rob R 0:00 1 deq2 360 debug run_mode rob R 0:00 1 deq2 235 debug vadeq_20 rob R 3:45:05 1 deq2

- And they complete in under 10 seconds, maybe 30 land segments times 40 land uses

squeue squeue: error: NodeNames=deq2 Sockets=0 is invalid, reset to 1 JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 235 debug vadeq_20 rob R 3:43:23 1 deq2

### Land Runoff Modeling
- this could be a real time saver
- Each set of 40+ land uses for a land segment are run in an `sbatch`, and 8 batches can run simultaneously.

segs=cbp get_landsegs P for i in $segs; do echo "sbatch /opt/model/meta_model/run_model hspf_cbp6 vadeq_2021 $i auto land " sbatch /opt/model/meta_model/run_model hspf_cbp6 vadeq_2021 $i auto land done

- Jobs complete without incident, with all indications that they are on time
  - `tail -f slurm-377.out`
  - Note: the messages below "No model plugins found in /opt/model/meta_model/models/hspf_cbp6/land/link/" is legit, there are no `link` or `analyze` plugins at this time.

tail -f slurm-377.out

running stf for segment N51610 land scenario vadeq_2021 '/opt/model/p6/vadeq/config/blank_wdm/land.wdm' -> 'stfN51610.wdm' count = 1 running sho for segment N51610 land scenario vadeq_2021 '/opt/model/p6/vadeq/config/blank_wdm/land.wdm' -> 'shoN51610.wdm' count = 1 No model plugins found in /opt/model/meta_model/models/hspf_cbp6/land/link/ No model plugins found in /opt/model/meta_model/models/hspf_cbp6/land/analyze/

[rburghol]

Needed met for Non VA P segments

• Find met scneario:
  • cbp get_config vadeq_2021 land MET
  • nldas1221
• Check met:
  • cbp check_met nldas1221 P

    Found 66 land-segments
    Missing land segment-landuse combos:
    N11001.wdm N24031.wdm N24033.wdm N51059.wdm N51510.wdm N51610.wdm N51600.wdm N51153.wdm N24017.wdm N51179.wdm N24037.wdm N51099.wdm N51193.wdm N51133.wdm N51061.wdm N51107.wdm N51683.wdm N51685.wdm N51013.wdm N24013.wdm N24021.wdm H24021.wdm N51069.wdm N51171.wdm N42001.wdm N24043.wdm N42055.wdm H51015.wdm N51015.wdm N51820.wdm H51165.wdm N51165.wdm N54031.wdm N51187.wdm N51043.wdm N51840.wdm N51660.wdm N42041.wdm N54003.wdm N54037.wdm N51139.wdm L51015.wdm N51790.wdm N42099.wdm N54065.wdm N42057.wdm H54031.wdm N54027.wdm N24001.wdm N42009.wdm H51139.wdm L54071.wdm N51091.wdm N54071.wdm H54023.wdm L54023.wdm N54023.wdm N54057.wdm H54071.wdm L54031.wdm N42111.wdm N24023.wdm H24023.wdm N54077.wdm N54093.wdm H54057.wdm

• But they DO exist in the p532 recent, so we have the ability to generate them:
  • fgrep N11001.wdm /files.dir
  • /opt/model/p53/p532c-sova/input/scenario/climate/met/nldas1221/met_N11001.wdm
  • ... and a whole bunch of others
• Generate the WDMs

  # generate phase 6 meteorology WDMs
  cd /opt/model/p6/vadeq/
  # do that subwatershed
  segs=`cbp get_landsegs P`
  for i in $segs; do
  #only need to create a WDM for each land seg
  # this script reads the file /etc/hspf.config to get directories.
  wdm_pm_one $i 1984010100 2020123123 nldas2 harp2021 nldas1221 p20211221
  done

Check the land

segs=`cbp get_landsegs P`
for i in $segs; do
  echo "sbatch /opt/model/meta_model/run_model hspf_cbp6 vadeq_2021 $i auto river"
  sbatch /opt/model/meta_model/run_model hspf_cbp6 vadeq_2021 $i auto river 
done

Run the land

segs=`cbp get_landsegs P`
for i in $segs; do
  echo "sbatch /opt/model/meta_model/run_model hspf_cbp6 vadeq_2021 $i auto land "
  sbatch /opt/model/meta_model/run_model hspf_cbp6 vadeq_2021 $i auto land 
done

[rburghol]

Headwater in Occoquan, PL2_4970_5250:

• Check if land has already run successfully
  • cbp basingen.csh vadeq_2021 4970
  • cbp check_land vadeq_2021 PL2_4970_5250
  • Looking for land segments: N51059 N51061 N51107 N51153 N51600 N51683 N51685
  • Found 43 land-uses, and 7 land-segments
  • Missing land segment-landuse combos:
  • NONE
• Since NONE is result of check_land, we can try to run this single segment (note: we do NOT need to run basingen.csh to run a single segment now)
  • /opt/model/meta_model/run_model hspf_cbp6 vadeq_2021 PL2_4970_5250 auto river

[rburghol]

Test:

• job is a short script that just pauses for 60 seconds

  
  for i in $segs; do
  deps=`cbp mm_river_deps $scenario $i`
  echo "Segment $i Found deps= $deps"
  job_name=`cbp mm_job_name $scenario $i`
  echo "sbatch --job-name=$job_name $deps /opt/model/meta_model/run_model hsp2_cbp6 $scenario $i  auto river"
  sbatch --job-name=$job_name $deps /opt/model/meta_model/run_model hsp2_cbp6 $scenario $i auto river
  done