Running and managing ePS remotely

phockett / epsman

Management tools for ePolyScat job generation & execution.

1 stars 2 forks source link

Running and managing ePS remotely #17

Open phockett opened 3 years ago

phockett commented 3 years ago

Current implementation (tries) to run shell script via Fabric, but this has issues with detached/background processes sometimes (see, e.g., start a background process with nohup using fabric and ssh TTY behaviour notes.

Todo:

[x] Fix current implementations? (As per start a background process with nohup using fabric, see Fabric user notes for more discussion). Now fixed in c7e91e4265619151c5f68f53eb966dadb7c54ec4.
[ ] Better/more robust implementations using local python process as manager? (Then run locally with subprocess.run() and/or os.fork() or similar - see, e.g., autonotebook code, esp triggerNotebook().)
[ ] Full job management routines on ePS machine? Python or via another toolset/manager/queue system, e.g. ts, CRON, supervisord, SLURM.... - see Fabric user notes for more discussion.
[ ] Wrapper for arb commands via Fabric.
[ ] Set & propagate custom run command for flexibility.
[ ] Use Docker? See https://github.com/phockett/open-photoionization-docker-stacks

Also/to debug:

[ ] Fix path setting for host=localhost only (i.e. running ePS on local machine). This currently may not be set, see https://github.com/phockett/epsman/blob/09782ca0d878a27705db1e5863c81cadb50c62bf/epsman/epsJob.py#L65

phockett commented 3 years ago

c7e91e4265619151c5f68f53eb966dadb7c54ec4 fixes ePS runner scripts and .runJobs() class method.

This implements runner as:

cmd = f" {self.hostDefn[self.host]['genFile'].as_posix()} &> /dev/null &"
result = self.c.run(Path(self.hostDefn[self.host]['scpdir'], 'ePS_batch_nohup.sh').as_posix() + cmd, 
                     warn = True, timeout = 10, pty=False)

For testing, set self.hostDefn[self.host]['genFile'] = Path(em.__path__[0]).parent/'notes'/'job-test.conf'.

Notes:

Basic solution from here, likely don't need nohup version of script here, although left as is for now.
Nohup log file now set to {genFile}.nohup.log in ePS_batch_nohup.sh, but may want to change this later.
No additional job monitoring or checking in this case, just fixing old scripts.

To do:

[ ] Further testing.
[ ] Test/demo notebook from scratch for a new machine.

UPDATE March 2022: see 5bdeecc25fff6b42dc1f5a56db848994ac58eb5a

phockett commented 1 year ago

Testing manual Docker runs on Fock cluster via Slurm.

e.g. srun -w fock-25 -l docker run -d --rm -v /software/docker/ePS_dockerBuildTests_2022:/data --env NCPUS=20 --name eps25-080523 fock:5000/eps:v040523 /data/runTests-fock-23.sh /data/tests-Fock-25-Slurm_test_080523

TODO:

[ ] Implement in docker run code
[ ] Add job tidy-up to run script (and/or fix users in container). Currently get files in completed output dirs belonging to nobody and missing permissions. See other Docker builds for ideas...
[ ] And/or update tidyJobs() functionality to do this.
[ ] Updated runJobs() for Slurm case.