search

Netflix / metaflow

Open Source Platform for developing, scaling and deploying serious ML, AI, and data science systems
https://metaflow.org
Apache License 2.0
8.09k stars 762 forks source link

Trigger flows on different kubernetes clusters #1624

Open omriel1 opened 11 months ago

omriel1 commented 11 months ago

Hi, I'm currently exploring Metaflow, specifically its functionality related to Triggering Flows Based on External Events, and I'm curious about its applicability to the following use case:

Assume I have two flows F1 and F2 and kubernetes clusters A and B. Ideally I'd want to be able to execute F1 on cluster A and once it finishes, make it trigger the execution of F2 on cluster B.

Can it be done in some way?

savingoyal commented 1 month ago

@omriel1 we now have support for multi-cluster execution within a single flow - where different steps can run on different clusters. that would be one option to achieve your use case - where flow B runs all the steps in a different cluster but is still deployed in the original kubernetes cluster. yet another option would be to configure a shared event bus for both the clusters . there are pros and cons of each approach (as well as few other approaches). let us know which direction you would like to explore.

omriel1 commented 1 month ago

Thank you for your response!