search

flyteorg / flyte

Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.
https://flyte.org
Apache License 2.0
5.4k stars 577 forks source link

[Plugin] Google Dataflow #818

Open kanterov opened 3 years ago

kanterov commented 3 years ago

Why would this plugin be helpful to the Flyte community Google Cloud Dataflow is a fully managed service for executing Apache Beam pipelines within the Google Cloud Platform ecosystem.

Type of Plugin

Can you help us with the implementation?

kanterov commented 3 years ago

There is some draft code, mainly implementation depends on token source mechanism introduced in https://github.com/flyteorg/flyte/issues/817

brucearctor commented 2 years ago

@Gui11aum3 is this something that exists and should be opened or that needs to be built. Please advise. As discussed, this seems a decent starting place.

Gui11aum3 commented 2 years ago

@brucearctor this hasn't been build. We thought we would have time to build it using the bigquery plugin as inspiration but never managed to get there :/ We could use it as a showcase on how to contribute to plugins.

brucearctor commented 2 years ago

Ya, this seems like one of the smaller/self-contained plugins which then is a better starting point.

Thanks for confirming!

brucearctor commented 2 years ago

ensure flags are part of spec. Ex: equivalent of --experiments=use_runner_v2 ...

brucearctor commented 2 years ago

See: https://github.com/flyteorg/flyte/discussions/2609

github-actions[bot] commented 1 year ago

Hello 👋, This issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will close the issue if we detect no activity in the next 7 days. Thank you for your contribution and understanding! 🙏

github-actions[bot] commented 1 year ago

Hello 👋, This issue has been inactive for over 9 months and hasn't received any updates since it was marked as stale. We'll be closing this issue for now, but if you believe this issue is still relevant, please feel free to reopen it. Thank you for your contribution and understanding! 🙏

github-actions[bot] commented 1 month ago

Hello 👋, this issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will engage on it to decide if it is still applicable. Thank you for your contribution and understanding! 🙏