search

payu-org / payu

A workflow management tool for numerical models on the NCI computing systems
Apache License 2.0
18 stars 25 forks source link

Add a debug option to payu-run #340

Open aidanheerdegen opened 1 year ago

aidanheerdegen commented 1 year ago

@harshula wanted to run a model directly, without payu running it.

I provided the following instructions:

  1. payu run and copy the PBS submission qsub line and cancel (^c) or delete the job (qdel)
  2. paste the PBS submission command (without the payu-run command) and append -I to run the command interactively and wait for the job to start
  3. once it has started run paste in the payu-run command from the PBS submit, e.g. g/data/hh5/public/apps/miniconda3/envs/analysis3-23.01/bin/python3.9 /g/data3/hh5/public/apps/miniconda3/envs/analysis3-23.01/bin/payu-run
  4. Wait until the mpirun command is printed to STDOUT cancel the job (^c) and copy and paste the actual mpirun command, e.g. 
    mpirun --mca io ompio --mca io_ompio_num_aggregators 1 -wdir /scratch/tm70/aph502/access-om2/work/1deg_jra55_iaf/atmosphere -np 1  /scratch/tm70/aph502/access-om2/work/1deg_jra55_iaf/atmosphere/yatm_5b6c697d.exe : -wdir /scratch/tm70/aph502/access-om2/work/1deg_jra55_iaf/
ocean -np 216  /scratch/tm70/aph502/access-om2/work/1deg_jra55_iaf/ocean/fms_ACCESS-OM_af3a94d4_libaccessom2_5b6c697d.x : -wdir /scratch/tm70/aph502/access-om2/work/1deg_jra55_iaf/ice -np 24  /scratch/tm70/aph502/access-om2/work/1deg_jra55_iaf/ice/cice_auscom_360x300_24p_2572851d_libaccessom2_5b6c697d.exe

These steps could be easily automated, which would be a useful service and make it really easy to run the configuration directly, for debugging purposes.

harshula commented 1 year ago

An option that generates a PBS script would be very useful. i.e. $ qsub -q <queue> -P <project> <pbs-script-generated-by-payu>

aidanheerdegen commented 1 year ago

An option that generates a PBS script would be very useful

Yes it would, but payu doesn't work that way. For example, payu run submits a PBS job that calls payu-run within it, and passes state information to the payu-run call through environment variables, so things like the current run number.

My thought was to just include a -I flag to make it interactive as a first basic step.

harshula commented 1 year ago

Is it technically possible for Payu to generate a PBS script?

aidanheerdegen commented 1 year ago

I guess it could but the environment information is passed in command line arguments, so it would require writing all that stuff to a script. I don't think this gives a much better outcome than an interactive session with the working directory populated and the relevant command printed to stdout, so you can copy and paste and run it directly. Plus it would require a lot more effort.