search

Netflix / metaflow

:rocket: Build and manage real-life ML, AI, and data science projects with ease!
https://metaflow.org
Apache License 2.0
7.79k stars 737 forks source link

ArgoEvents.publish attempts to make requests even when url is None #1903

Open emilyselwood opened 6 days ago

emilyselwood commented 6 days ago

We have been seeing a bunch of error messages at the ends of our steps like:

Unable to publish Argo Event (metaflow.ExportFlow.start): unknown url type: 'None'

These were happening at the same time as other errors so we spent a bunch of time figuring out what they meant. I think I've tracked down to the publish method in argo_events. We have not set the ARGO_EVENTS_WEBHOOK_URL so None is being passed along to the request which fails.

https://github.com/Netflix/metaflow/blob/39524f7cddf62946bc4f032cc050f305b2d9d595/metaflow/plugins/argo/argo_events.py#L125

My guess is a check could be added to the function that forces an early return if self._url is None, preventing it attempting to make a http request that will fail. I'd be happy to make that change if people agree that this is whats happening?

savingoyal commented 6 days ago

Thanks for opening the issue. Silently returning can mask config-related issues where the user expects an event to be emitted. A work-around in your case could be to submit the flows to argo-workflows using the --no-auto-emit-argo-events flag. Would that work?