Netflix / metaflow

:rocket: Build and manage real-life ML, AI, and data science projects with ease!
https://metaflow.org
Apache License 2.0
7.79k stars 737 forks source link

bugfix: properly deletes Argo Events trigger sensors when @project is used #1871

Closed gabriel-rp closed 4 weeks ago

gabriel-rp commented 1 month ago

This fixes issue #1870.

Tests

I did not find any test setup to create unit or integration tests. So my only test was replacing the Metaflow library with my fork and ensuring the sensors would be deleted. Before the change, they were not.

Here are the logs for the same flow used in the issue description. As expected, this time the sensors were deleted. I have also checked the Kubernetes resources.

Metaflow 2.12.0 executing MyTestFlow for user:g.pereira.3
Project: my_test_project, Branch: user.g.pereira.3
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
Deleting workflow mytestproject.user.g.pereira.3.mytestflow...
Deleting sensor mytestproject.user.g.pereira.3.mytestflow...
Deleting Kubernetes resources may take a while. Deploying the flow again to Argo Workflows while the delete is in-flight will fail.
In-flight executions will not be affected. If necessary, terminate them manually.

@project(name="my_test_project") @trigger(event="test_airflow_trigger") class MyTestFlow(FlowSpec): @step def start(self): self.next(self.end)

@step
def end(self):
    pass

if name == "main": MyTestFlow()