Save Dagster state to S3

datonic / datadex

📦 Serverless and local-first Open Data Platform

http://datadex.datonic.io

MIT License

220 stars 14 forks source link

Closed davidgasquez closed 1 month ago

davidgasquez commented 3 months ago

Can save the SQLite database. Useful if we want to keep the state. Not very urgent for now as we're reconstructing everything with each run.

I'm also personally laying more into using the DuckDB database itself as the state and recover from it.

davidgasquez commented 1 month ago

Closing as I think I'd keep the sate at the DuckDB level when needed.