Closed davidgasquez closed 6 months ago
New release notes.
When a Nothing-typed output is returned from an asset or op, the handle_output function of the I/O manager will no longer be called.
I wonder if this means -> Optional[pd.DataFrame]
return from an asset could now cause to I/O manager being used only, when there is some data to be saved.
Going to check this, although correct solution is likely to just disable I/O managers entirely in favour of resources.
:wave: I've given this a try and complicates a bit the code (adding boilerplate). Let's keep em for the moment on the small assets. For the incremental and larger assets, we can use the duckdb
resource.
After talking with some Dagster employees and looking at Dagster own project, feels like moving out of IO Manager will make it easier to create both sensors and partitioned assets for #28.