Open davidgasquez opened 8 months ago
Dune has an upload limit of 1MiB...
We can upload probably some daily metrics but not much more.
@DistributedDoge, do you know how the Flipside external data works?
Seems their live query stuff would work for JSON data but I'm not sure if it can read Parquet files
From what I see in docs+UI Dune is telling me upload limit is 200MB per .CSV table. See here, no clue if it is accurate though.
Last time I tried, Flipside could not read anything other than JSON, it will most certainly work with ipfs-hosted file though.
EDIT: For 2, to avoid making things messy we could e.g. push a subset of tables as json
to different Filebase bucket.
Dune is telling me upload limit is 200MB per .CSV table
I think that might be correct. The limit comes from the Free plan, which limits storage to 1MB.
For 2, to avoid making things messy we could e.g. push a subset of tables as json to different Filebase bucket.
That could work! We'll need to check how efficient those queries are though. Querying uncompressed JSON is not very efficient.
Confirmed empirically that Dune does not let me upload 25MB file, bait and switch at its finest.
They're thinking about increasing it though.
I can poke them a bit more and check if that's a "future" plan or actual thing happening in the following weeks/months.
I just learned we can in fact select assets using groups, compute kind or other properties.
This means we can probably construct assset job definitions in such a way, that say CI run
would only trigger default
assets while weekly fetch
would trigger everything including actually updating Dune or Flipside.
from dagster import AssetSelection
AssetSelection.groups('group_label')
Sounds like the perfect approach!
For Flipside, a new option can be to push data to a spreadsheet and have Flipside read from it.
Seems like Dune live fetch is now open to all free users!
We can push datasets to Flipside and Dune via Dagster assets.