Tioga uses Flux to schedule jobs and has limited support for srun. Also Tioga does not have the scontrol command, which we use to retrieve the allocated node list.
This PR includes native support for Flux. It uses flux run to run clients and servers and uses flux resource to retrieve the number of nodes and the node list.
PS: flux resource returns a condensed node list, e.g., tioga[3-10, 12, 14]. The existing parse_hostfile() function can't handle this format, I added some code to parse it manually.
How Has This Been Tested?
Tested on Tioga with 1, 2, 4 nodes. Also tested unifyfs-ls, unifyfs-stage and stage-in/out features.
Types of changes
[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Performance enhancement (non-breaking change which improves efficiency)
[ ] Code cleanup (non-breaking change which makes code smaller or more readable)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Testing (addition of new tests or update to current tests)
[ ] Documentation (a change to man pages or other documentation)
Checklist:
[x] My code follows the UnifyFS code style requirements.
[ ] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes.
[x] All new and existing tests passed.
[x] All commit messages are properly formatted.
TODO
Unlike slurm where SLURM_JOBID can be used to determine a slurm allocation, flux only sets environment variables such as FLUX_JOB_ID for each flux run job (a flux job is similar to a slurm step). At the time of executing unifyfs (batch level), those variables have not been set yet.
A short flux script example:
#!/bin/bash
# flux: -N4 -n256 -t 5m
# flux: --job-name="UnifyFS"
# flux: --queue="pdebug"
export UNIFYFS_LOGIO_SPILL_DIR=/tmp
export UNIFYFS_LOG_DIR=`pwd`
export UNIFYFS_LOG_VERBOSITY=3
export UNIFYFS_MOUNTPOINT=/unifyfs
# Here FLUX_JOB_ID are FLUX_JOB_NNODES are not set
unifyfs start -d --stage-in=`pwd`/manifest-in.txt --share-dir=`pwd`
flux run -n128 -N4 -c1 $UNIFYFS_DIR/libexec/write-static -m /unifyfs -f myTestFile
flux run -n128 -N4 -c1 $UNIFYFS_DIR/libexec/write-gotcha -f workflowTestFile
flux run -n128 -N4 -c1 $UNIFYFS_DIR/libexec/read-gotcha -f workflowTestFile
flux run -n16 -N4 -c1 unifyfs-stage --parallel --status-file=/tmp/stage-out-status.txt `pwd`/manifest-out.txt
unifyfs-ls
unifyfs terminate --cleanup
As a result, currently I use FLUX_EXEC_PATH to determine if the system has flux scheduler. I feel this is not optimal but I couldn't figure out a better way.
adammoody
commented
11 months ago
Description
Tioga uses Flux to schedule jobs and has limited support for srun. Also Tioga does not have the
scontrolcommand, which we use to retrieve the allocated node list.This PR includes native support for Flux. It uses
flux runto run clients and servers and usesflux resourceto retrieve the number of nodes and the node list. PS:flux resourcereturns a condensed node list, e.g., tioga[3-10, 12, 14]. The existingparse_hostfile()function can't handle this format, I added some code to parse it manually.How Has This Been Tested?
Tested on Tioga with 1, 2, 4 nodes. Also tested
unifyfs-ls,unifyfs-stageand stage-in/out features.Types of changes
Checklist:
TODO
Unlike slurm where
SLURM_JOBIDcan be used to determine a slurm allocation, flux only sets environment variables such asFLUX_JOB_IDfor eachflux runjob (a flux job is similar to a slurm step). At the time of executingunifyfs(batch level), those variables have not been set yet.A short flux script example:
As a result, currently I use
FLUX_EXEC_PATHto determine if the system has flux scheduler. I feel this is not optimal but I couldn't figure out a better way.