search

lanl / BEE

Other
14 stars 3 forks source link

Add pre/post script support #788

Closed rstyd closed 6 months ago

rstyd commented 7 months ago

This PR addresses #773. It adds pre/post script support through a beeflow:ScriptRequirement hint which has the format:

    beeflow:ScriptRequirement:
    enabled: true
    pre_script_path: "pre_run.sh"
    post_script_path: "post_run.sh

The pre_run and post_run scripts are read using the parser and passed to the task object for the associated task. It's then read and added to the SBATCH script in the slurm runner.

pagrubel commented 7 months ago

@rstyd Your example works. Please add it to all the workers and can you set up a test in CI? I think there should be a way to att it to the Flux and Slurm tests using a simpler example.

pagrubel commented 6 months ago
  1. Add pre/post to all workers.
  2. Add test to slurm and flux integration test.
  3. Documentation.
pagrubel commented 6 months ago

@kchilleri There may be some misunderstanding. Yes the flux implementation should be separate. However, this PR is still missing documentation and integration tests. So really it shouldn't be approved as is.

kchilleri commented 6 months ago

@pagrubel Got it! Should I dismiss my approval?

pagrubel commented 6 months ago

@rstyd Please squash the commits, or I can if you wish.