i-VRESSE / bartender

Middleware web service to schedule jobs on various infrastructures
https://i-vresse-bartender.readthedocs.io
Apache License 2.0
1 stars 0 forks source link

Written Dirac scheduler / filesystem #47

Closed sverhoeven closed 1 year ago

sverhoeven commented 1 year ago

Refs #38

TODO

  1. [x] job submission
  2. [x] job input from local to grid storage
  3. [x] job input from grid storage to compute site
  4. [x] job output from compute site to grid storage
  5. [x] job output from grid storage to local
  6. [x] dirac test suite
  7. [x] dirac test suite in CI
  8. [x] dev container
  9. [x] clean up commented out code
  10. [x] fix TODOs in code or convert to issues
  11. [x] make logging less verbose
  12. [x] get CI green
  13. [x] make bartender work without DIRAC installed
  14. [x] move tests for dirac /tests-dirac/test*.py inside /tests and use pytest markers to run non-dirac tests by default and dirac tests when specified.
  15. [x] Handle when proxy has expired, in test container valid for 24 hours.

This PR adds the bartender.filesystems.dirac.DiracFileSystem and bartender.schedulers.dirac.DiracScheduler classes. To test them you

sverhoeven commented 1 year ago

Thanks for reviewing.

There are 2 types of auth in bartender.

  1. To consume the web service you need auth
  2. Some destinations need auth, like dirac or slurm over ssh. A job submitted via the web services requires auth, but job is run by destination user/proxy on destination machine.