search

Netflix / metaflow

Open Source Platform for developing, scaling and deploying serious ML, AI, and data science systems
https://metaflow.org
Apache License 2.0
8.29k stars 775 forks source link

Trigger flows on different kubernetes clusters #1624

Open omriel1 opened 1 year ago

omriel1 commented 1 year ago

Hi, I'm currently exploring Metaflow, specifically its functionality related to Triggering Flows Based on External Events, and I'm curious about its applicability to the following use case:

Assume I have two flows F1 and F2 and kubernetes clusters A and B. Ideally I'd want to be able to execute F1 on cluster A and once it finishes, make it trigger the execution of F2 on cluster B.

Can it be done in some way?

savingoyal commented 3 months ago

@omriel1 we now have support for multi-cluster execution within a single flow - where different steps can run on different clusters. that would be one option to achieve your use case - where flow B runs all the steps in a different cluster but is still deployed in the original kubernetes cluster. yet another option would be to configure a shared event bus for both the clusters . there are pros and cons of each approach (as well as few other approaches). let us know which direction you would like to explore.

omriel1 commented 3 months ago

Thank you for your response!