i-VRESSE / bartender

Middleware web service to schedule jobs on various infrastructures
https://i-vresse-bartender.readthedocs.io
Apache License 2.0
1 stars 0 forks source link

Multi scheduler support #17

Closed sverhoeven closed 1 year ago

sverhoeven commented 1 year ago

Uses configuration file to specify applications, schedulers and filesystems. By default uses first scheduler and first filesystem to upload input, submit, state fetch, download output.

To test the multi scheduler support you have to create a config.yaml file by copying config-example.yaml. The example config file uses a in-memory scheduler on the local file system. To use the Slurm scheduler and sftp filesystem

  1. you have to uncomment the slurmcontainer block and
  2. start up a Slurm Docker container as described in the config file.
  3. you should be able to submit a job that
    1. Copies input files to container in /home/xenon/<job_id>, which you can verify with docker exec <container id> ls -l /home/xenon/<job_id>
    2. Submit job to Slurm, which you can verify by running sacct to see the completed job.
    3. After job completion will copy the output files back to where bartender serve is running, which you can verify with ls /tmp/jobs/*, it should have a stdout.txt file.

Also