Bug: Graph.from_tasks with flow.tasks.values() raises "RuntimeError: dictionary changed size during iteration"

[lieutdan13]

The controller.graph property raises a Runtime: dictionary changed size during iteration exception.

I've only been able to replicate this with task B depending on task A. However, I suspect that if the flow adds subtasks while iterating over the current tasks, this error is raised.

A simple solution may be to caste the self.flow.tasks.values() to a list before calling Graph.from_tasks.

Patch will be submitted soon.

[jlowin]

@lieutdan13 thanks for the report -- we've just merged a sizable refactor that, among other things, removes Graph.from_tasks and changes how graphs are instantiated. I hope it resolves your issue. However if it doesn't please create an MRE and I'm sure we can quickly address it -- your logic sound correct to me, so it's a situation we want to make sure nobody gets stuck in!

[MarkEdmondson1234]

I got this too today:

Task run failed with exception: RuntimeError('Set changed size during iteration') - Retries are exhausted
Traceback (most recent call last):
  File "/Users/mark/dev/sunholo/control-flow/.venv/lib/python3.12/site-packages/prefect/task_engine.py", line 673, in run_context
    yield self
  File "/Users/mark/dev/sunholo/control-flow/.venv/lib/python3.12/site-packages/prefect/task_engine.py", line 729, in run_task_sync
    engine.call_task_fn(txn)
  File "/Users/mark/dev/sunholo/control-flow/.venv/lib/python3.12/site-packages/prefect/task_engine.py", line 702, in call_task_fn
    result = call_with_parameters(self.task.fn, parameters)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mark/dev/sunholo/control-flow/.venv/lib/python3.12/site-packages/prefect/utilities/callables.py", line 208, in call_with_parameters
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/mark/dev/sunholo/control-flow/.venv/lib/python3.12/site-packages/controlflow/controllers/controller.py", line 270, in run_once
    payload = self._setup_run()
              ^^^^^^^^^^^^^^^^^
  File "/Users/mark/dev/sunholo/control-flow/.venv/lib/python3.12/site-packages/controlflow/controllers/controller.py", line 203, in _setup_run
    instructions = instructions_template.render()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mark/dev/sunholo/control-flow/.venv/lib/python3.12/site-packages/controlflow/controllers/instruction_template.py", line 233, in render
    g = self.controller.graph
        ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mark/dev/sunholo/control-flow/.venv/lib/python3.12/site-packages/controlflow/controllers/controller.py", line 93, in graph
    return Graph.from_tasks(self.flow.tasks.values())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mark/dev/sunholo/control-flow/.venv/lib/python3.12/site-packages/controlflow/controllers/graph.py", line 59, in from_tasks
    graph.add_task(task)
  File "/Users/mark/dev/sunholo/control-flow/.venv/lib/python3.12/site-packages/controlflow/controllers/graph.py", line 75, in add_task
    for upstream in task.depends_on:
RuntimeError: Set changed size during iteration