Closed davidgasquez closed 1 month ago
Chatted with some folks working on Subsquid. They're doing interesting things on the decentralized data lake area.
This is more or less what I understood about how the Subsquid Archive works.
substrate-ingest
). In the future, anyone will be able to publish their arbitrary datasets.Subsquid Labs maintains public Archive endpoints and offers batch access via the Squid SDK free of charge.
Adding a small note that Dagster is already relying on "hashes" to check when runs are needed! A step closer to fully content addresses workflows.
You can ATTACH
to a remote DuckDB database! There might be a world where a bunch of people publish their small/medium databases and people just attach to them.
Random thoughts around decentralized and permissionless data lakes.
Also from https://github.com/davidgasquez/datadex/issues/22#issuecomment-1558988508.