dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
11.66k stars 1.47k forks source link

avoid calling `handle_output` twice per Dagstermill output #12830

Open sryza opened 1 year ago

sryza commented 1 year ago

If someone yields an Output from within their Dagstermill notebook, the following happens in sequence:

The reason we do this is that we need to bubble up an Output to the step worker framework code to let it know that an output was yielded, so it can write out the appropriate events to the instance.

One way to fix this would be to offer a way for the dagstermill code to say "trust me, I'm handling the output" and then have the framework code write out the appropriate events without directly touching the output object.

rahuldave commented 1 year ago

This sounds like a great idea!