search

CliMA / slurm-buildkite

Run buildkite jobs on a slurm cluster
Other
7 stars 0 forks source link

Move pipeline.yml file to repository #2

Closed simonbyrne closed 4 years ago

simonbyrne commented 4 years ago

https://buildkite.com/docs/pipelines/defining-steps#adding-steps-pipeline-dot-yml-file

jakebolewski commented 4 years ago

we might want use the script generation option (generate the yaml with a script)

that would cut down a lot on the boilerplate and we could re-use the definitions that already exist (slurm-tests.toml)

simonbyrne commented 4 years ago

we might be able to just move the boilerplate to a setup script, e.g. so a single job would just be

- name: "jobname"
  commands:
    - central-gpu-setup.sh
    - mpiexec julia --project experiments/...
  agents:
    - ntasks: 4
    - gres: "gpu:4"
jakebolewski commented 4 years ago

that's probably easier and less magical

simonbyrne commented 4 years ago

Or we could do crazy things with https://github.com/marketplace/actions/trigger-buildkite-pipeline

simonbyrne commented 4 years ago

done