Closed MSamman closed 1 year ago
Thanks @MSamman for contributing this PR! I'm excited to have GCP support for redun. I'll start reviewing and giving feedback.
@mattrasmus - this PR doesn't yet support file staging. However, I have implemented staging support in two ways.
(1) Staging via mounting
https://github.com/msamman/redun/tree/gcp-mount2
(2) Staging via gsutil
https://github.com/msamman/redun/tree/gcp-stage
Of the two, I think staging via mounting is much cleaner. I imagine there are improvements that can be made with either approach. Happy to discuss further how to move forward on this.
I have merged the mounting approach to start.
Nice PR! @MSamman @danielecook , we started using the gcp batch implementation internally and @tfehlmann added some nice features (e.g. transfer with gcloud storage
and automatic machine selection a la nextflow) . Let us know when you're ready to merge and we can integrate them. The changes live here: https://github.com/retrobiosciences/redun
Nice PR! @MSamman @danielecook , we started using the gcp batch implementation internally and @tfehlmann added some nice features (e.g. transfer with
gcloud storage
and automatic machine selection a la nextflow) . Let us know when you're ready to merge and we can integrate them. The changes live here: https://github.com/retrobiosciences/redun
Awesome! I am happy to see there is interest in a GCP batch executor.
I am curious about your gcloud staging implementation:
I am also curious how your implementation is working in practice. I think there are still a few details here to hammer out.
Closing in favor #68
Thanks @MSamman and @danielecook for contribution! I've merged your changed in #68.
Work in progress for adding GCP batch support. Currently can only submit single jobs.
Looking for feedback! Thanks!