search

Netflix / metaflow

Open Source Platform for developing, scaling and deploying serious ML, AI, and data science systems
https://metaflow.org
Apache License 2.0
8.25k stars 774 forks source link

Error on argo-workflows create #2013

Closed colebaileygit closed 2 months ago

colebaileygit commented 2 months ago

When running command ... argo-workflows create I get this error when using metaflow 2.12.17 or 2.12.18 but not when using older versions like 2.10.3:

    Internal error
Traceback (most recent call last):
  File "/Users/c.bailey/Source/logistics-ds-dummy-project/.venv/lib/python3.11/site-packages/metaflow/cli.py", line 1134, in main
    start(auto_envvar_prefix="METAFLOW", obj=state)
  File "/Users/c.bailey/Source/logistics-ds-dummy-project/.venv/lib/python3.11/site-packages/metaflow/tracing/__init__.py", line 27, in wrapper_func
    return func(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/c.bailey/Source/logistics-ds-dummy-project/.venv/lib/python3.11/site-packages/metaflow/_vendor/click/core.py", line 829, in __call__
    return self.main(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/c.bailey/Source/logistics-ds-dummy-project/.venv/lib/python3.11/site-packages/metaflow/_vendor/click/core.py", line 782, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/c.bailey/Source/logistics-ds-dummy-project/.venv/lib/python3.11/site-packages/metaflow/_vendor/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/c.bailey/Source/logistics-ds-dummy-project/.venv/lib/python3.11/site-packages/metaflow/_vendor/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/c.bailey/Source/logistics-ds-dummy-project/.venv/lib/python3.11/site-packages/metaflow/_vendor/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/c.bailey/Source/logistics-ds-dummy-project/.venv/lib/python3.11/site-packages/metaflow/_vendor/click/core.py", line 610, in invoke
    return callback(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/c.bailey/Source/logistics-ds-dummy-project/.venv/lib/python3.11/site-packages/metaflow/_vendor/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/c.bailey/Source/logistics-ds-dummy-project/.venv/lib/python3.11/site-packages/metaflow/plugins/argo/argo_workflows_cli.py", line 257, in create
    flow = make_flow(
           ^^^^^^^^^^
  File "/Users/c.bailey/Source/logistics-ds-dummy-project/.venv/lib/python3.11/site-packages/metaflow/plugins/argo/argo_workflows_cli.py", line 486, in make_flow
    return ArgoWorkflows(
           ^^^^^^^^^^^^^^
  File "/Users/c.bailey/Source/logistics-ds-dummy-project/.venv/lib/python3.11/site-packages/metaflow/plugins/argo/argo_workflows.py", line 174, in __init__
    self._workflow_template = self._compile_workflow_template()
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/c.bailey/Source/logistics-ds-dummy-project/.venv/lib/python3.11/site-packages/metaflow/plugins/argo/argo_workflows.py", line 858, in _compile_workflow_template
    .templates(self._dag_templates())
               ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/c.bailey/Source/logistics-ds-dummy-project/.venv/lib/python3.11/site-packages/metaflow/plugins/argo/argo_workflows.py", line 1280, in _dag_templates
    for daemon_template in self._daemon_templates()
                           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/c.bailey/Source/logistics-ds-dummy-project/.venv/lib/python3.11/site-packages/metaflow/plugins/argo/argo_workflows.py", line 2157, in _daemon_templates
    templates.append(self._heartbeat_daemon_template())
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/c.bailey/Source/logistics-ds-dummy-project/.venv/lib/python3.11/site-packages/metaflow/plugins/argo/argo_workflows.py", line 2583, in _heartbeat_daemon_template
    start_step = next(step for step in self.flow if step.name == "start")
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
StopIteration
colebaileygit commented 2 months ago

Latest version that works for me is 2.12.12 with 2.12.13 being the first to break

savingoyal commented 2 months ago

@colebaileygit can you share a reproducible example?

savingoyal commented 2 months ago

@colebaileygit the newest release of metaflow shouldn't have this issue anymore. #1999