radical-cybertools / radical.pilot

RADICAL-Pilot
http://radical-cybertools.github.io/radical-pilot/index.html
Other
54 stars 23 forks source link

Create 'lite' launcher for RP #3099

Open mtitov opened 9 months ago

mtitov commented 9 months ago

Proposed launcher will make a batch submission with RP script as a target application.

(*) PR for https://github.com/radical-cybertools/radical.pilot/issues/3094 should be in place

mtitov commented 9 months ago

This is how it will look for the user: radical-pilot-launcher rp_app.py -c job_descr.json (where job_descr.json includes project, queue, runtime, etc.)

The same job_descr should be used by RP for the pilot description when it starts within the batch job. Thus, user doesn't need to keep this information in sync between json file and PilotDescription inside rp_app.py, and RP will know which one should be used.

(*) one of the options is to use env variables for the provided config json file, keep them as part of the batch script, and PilotDescription will use them instead of ones that are set (the case when RP determines that it started within the batch job, it can look for a corresponding env variables)

mtitov commented 7 months ago

Launcher to run IMPECCABLE workflow(s) on Frontier

```shell #!/bin/bash WORK_DIR=$PWD/runinfo if ! test -d "${WORK_DIR}"; then mkdir -p "${WORK_DIR}" fi BASE_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]:-$0}"; )" &> /dev/null && pwd 2> /dev/null; )" CONFIG=$(cat "${BASE_DIR}"/config.json) ACCOUNT=$(echo "${CONFIG}" | jq -r .run_description.project) PARTITION=$(echo "${CONFIG}" | jq -r .run_description.queue) NNODES=$(echo "${CONFIG}" | jq .run_description.nodes) RUNTIME=$(echo "${CONFIG}" | jq .run_description.walltime) CORE_SPEC=$(echo "${CONFIG}" | jq .node_description.core_spec) SMT=$(echo "${CONFIG}" | jq .node_description.smt) PROJ_HOME_DIR=$(echo "${CONFIG}" | jq -r .env.PROJ_HOME_DIR) while getopts "m:n:t:" OPTION; do case $OPTION in m) MODULE="$OPTARG" ;; n) NNODES="$OPTARG" ;; t) RUNTIME="$OPTARG" ;; *) exit 1 ;; esac done if ! test -z "$MODULE"; then OPT_MODULE="-m $MODULE" fi OPT_NNODES="-n $NNODES" OPT_RUNTIME="-t $RUNTIME" cat > /tmp/$$.slurm < $WORK_DIR/run_report.txt 2>&1