davidgasquez / gitcoin-grants-data-portal

🌲 Open source, serverless, and local-first data hub for Gitcoin Grants data!
https://grantsdataportal.xyz/
MIT License
26 stars 3 forks source link

Migrate out of Dagster IO Managers #32

Closed davidgasquez closed 6 months ago

davidgasquez commented 8 months ago

After talking with some Dagster employees and looking at Dagster own project, feels like moving out of IO Manager will make it easier to create both sensors and partitioned assets for #28.

DistributedDoge commented 7 months ago

New release notes.

When a Nothing-typed output is returned from an asset or op, the handle_output function of the I/O manager will no longer be called.

I wonder if this means -> Optional[pd.DataFrame] return from an asset could now cause to I/O manager being used only, when there is some data to be saved.

Going to check this, although correct solution is likely to just disable I/O managers entirely in favour of resources.

davidgasquez commented 6 months ago

:wave: I've given this a try and complicates a bit the code (adding boilerplate). Let's keep em for the moment on the small assets. For the incremental and larger assets, we can use the duckdb resource.