If someone yields an Output from within their Dagstermill notebook, the following happens in sequence:
Within the notebook process:
We invoke handle_output to write it to persistent storage
Within the Dagster step process that launched the notebook process:
We invoke load_input to read it from persistent storage
We invoke handle_output to write it to persistent storage
The reason we do this is that we need to bubble up an Output to the step worker framework code to let it know that an output was yielded, so it can write out the appropriate events to the instance.
One way to fix this would be to offer a way for the dagstermill code to say "trust me, I'm handling the output" and then have the framework code write out the appropriate events without directly touching the output object.
If someone yields an
Output
from within their Dagstermill notebook, the following happens in sequence:handle_output
to write it to persistent storageload_input
to read it from persistent storagehandle_output
to write it to persistent storageThe reason we do this is that we need to bubble up an
Output
to the step worker framework code to let it know that an output was yielded, so it can write out the appropriate events to the instance.One way to fix this would be to offer a way for the dagstermill code to say "trust me, I'm handling the output" and then have the framework code write out the appropriate events without directly touching the output object.