search

insitro / redun

Yet another redundant workflow engine
https://insitro.github.io/redun/
Apache License 2.0
510 stars 43 forks source link

GCP batch support #66

Closed MSamman closed 1 year ago

MSamman commented 1 year ago

Work in progress for adding GCP batch support. Currently can only submit single jobs.

Looking for feedback! Thanks!

mattrasmus commented 1 year ago

Thanks @MSamman for contributing this PR! I'm excited to have GCP support for redun. I'll start reviewing and giving feedback.

danielecook commented 1 year ago

@mattrasmus - this PR doesn't yet support file staging. However, I have implemented staging support in two ways.

(1) Staging via mounting

https://github.com/msamman/redun/tree/gcp-mount2

(2) Staging via gsutil

https://github.com/msamman/redun/tree/gcp-stage

Of the two, I think staging via mounting is much cleaner. I imagine there are improvements that can be made with either approach. Happy to discuss further how to move forward on this.

danielecook commented 1 year ago

I have merged the mounting approach to start.

ricomnl commented 1 year ago

Nice PR! @MSamman @danielecook , we started using the gcp batch implementation internally and @tfehlmann added some nice features (e.g. transfer with gcloud storage and automatic machine selection a la nextflow) . Let us know when you're ready to merge and we can integrate them. The changes live here: https://github.com/retrobiosciences/redun

danielecook commented 1 year ago

Nice PR! @MSamman @danielecook , we started using the gcp batch implementation internally and @tfehlmann added some nice features (e.g. transfer with gcloud storage and automatic machine selection a la nextflow) . Let us know when you're ready to merge and we can integrate them. The changes live here: https://github.com/retrobiosciences/redun

Awesome! I am happy to see there is interest in a GCP batch executor.

I am curious about your gcloud staging implementation:

  1. Do you package gcloud in containers?
  2. How do you handle permissions?

I am also curious how your implementation is working in practice. I think there are still a few details here to hammer out.

mattrasmus commented 1 year ago

Closing in favor #68

Thanks @MSamman and @danielecook for contribution! I've merged your changed in #68.