"__ASSET_JOB_1 has no op named" error with asset reconciliation sensor and run retries

[sryza]

When an automatic run retry is launched for a run launched by the asset reconciliation sensor, sometimes it hits an error like this:

dagster._check.CheckError: Invariant failed. Description: __ASSET_JOB_1 has no op named kippcamden__powerschool__assignmentscore.

Reported twice:

• https://elementl-workspace.slack.com/archives/C03VBNRRW4F/p1680538512239709
• https://dagster.slack.com/archives/C01U954MEER/p1680539022668849

[mgierada]

Yeah I had the same issue! The quick fix for me is to disable automatic retries through Dagster Cloud UI.

[sryza]

I was finally able to reproduce this. Minimal steps:

• Two assets, one depends on the other, the second one always fails
• Run retries configured
• Materialize the first one
• Materialize the second one (and observe failure)
• Run retry for the second one should fail with the error listed above

It appears that source of the problem is that the execution plan for the retry ends up containing all steps, instead of just the step that should be retried.

Interestingly, the same error does not occur when launching a manual re-execution from failure.

Here's what gets passed to create_run in the auto-retry case (which is where the problem is):

asset_selection: frozenset({AssetKey(['multi_dynamic_downstream2'])})
solid_selection: None
execution_plan_snapshot.steps: ['multi_dynamic_downstream2', 'multi_dynamic_upstream2', 'non_partitioned_asset']
pipeline_snapshot.node_names: ['multi_dynamic_downstream2', 'multi_dynamic_upstream2', 'non_partitioned_asset']

Here's what gets passed to create_run in the manual retry case (which works correctly):

asset_selection: frozenset({AssetKey(['multi_dynamic_downstream2'])})
solid_selection: None
execution_plan_snapshot.steps: ['multi_dynamic_downstream2']
pipeline_snapshot.node_names: ['multi_dynamic_downstream2']

[mgierada]

"It appears that source of the problem is that the execution plan for the retry ends up containing all steps, instead of just the step that should be retried."

I totally agree with that! Thanks @sryza for digging in!