search

insitro / redun

Yet another redundant workflow engine
https://insitro.github.io/redun/
Apache License 2.0
520 stars 45 forks source link

Google cloud #34

Closed jshleap closed 1 year ago

jshleap commented 2 years ago

Hi! this is quite a clever pythonic and readable workflow manager! thank you for it.

If I understand correctly, your cloud support is limited to AWS (AWS Batch, AWS Glue). Any plans to add GCP support?

mattrasmus commented 2 years ago

Hi @jshleap, thanks for question.

We currently support all the object storage systems provided by fsspec, which include GCS. As for compute, the executor design should allow many different backends over time. The closest to executor to being useful in GCP might be the kubernetes executor being worked on in #22. Let me know if that helps.

jshleap commented 2 years ago

Not quite, unless image can be a GCP uri? The ideal scenario would be that given an image and an storage uris, you could run "batch" in either of the cloud providers. But I understand that it could mean a lot of work. Thanks for the answer.

mattrasmus commented 2 years ago

Ah, great point. The Docker repository would need to be generalized as well.

So far we've been prioritizing work as it comes up naturally in our own projects, but PRs are welcomed as well. Thanks again for the ideas.

mattrasmus commented 2 years ago

Just to give an update here. Our docker uri can be any docker registry, so there isn't anything ECR specific.

We also have generalized our File staging to support GCS and other fsspec supported filesystem.

We are still working on #22 for providing an actual executor.

danielecook commented 2 years ago

@mattrasmus I'm curious if anyone is currently working on implementing a Google Batch executor for redun.

jspaezp commented 1 year ago

I am adding this here as a reference https://github.com/insitro/redun/pull/66 Seems like GCP batch has already been added