Flipside/Dune Integration

[davidgasquez]

We can push datasets to Flipside and Dune via Dagster assets.

[davidgasquez]

Dune has an upload limit of 1MiB...

We can upload probably some daily metrics but not much more.

@DistributedDoge, do you know how the Flipside external data works?

Seems their live query stuff would work for JSON data but I'm not sure if it can read Parquet files

[DistributedDoge]

1. From what I see in docs+UI Dune is telling me upload limit is 200MB per .CSV table. See here, no clue if it is accurate though.

2. Last time I tried, Flipside could not read anything other than JSON, it will most certainly work with ipfs-hosted file though.

EDIT: For 2, to avoid making things messy we could e.g. push a subset of tables as json to different Filebase bucket.

[davidgasquez]

Dune is telling me upload limit is 200MB per .CSV table

I think that might be correct. The limit comes from the Free plan, which limits storage to 1MB.

For 2, to avoid making things messy we could e.g. push a subset of tables as json to different Filebase bucket.

That could work! We'll need to check how efficient those queries are though. Querying uncompressed JSON is not very efficient.

[DistributedDoge]

Confirmed empirically that Dune does not let me upload 25MB file, bait and switch at its finest.

[davidgasquez]

They're thinking about increasing it though.

I can poke them a bit more and check if that's a "future" plan or actual thing happening in the following weeks/months.

[davidgasquez]

Left a comment with a potential implementation approach.

[DistributedDoge]

I just learned we can in fact select assets using groups, compute kind or other properties.

This means we can probably construct assset job definitions in such a way, that say CI run would only trigger default assets while weekly fetch would trigger everything including actually updating Dune or Flipside.

from dagster import AssetSelection
AssetSelection.groups('group_label')

[davidgasquez]

Sounds like the perfect approach!

[davidgasquez]

For Flipside, a new option can be to push data to a spreadsheet and have Flipside read from it.

[davidgasquez]

Seems like Dune live fetch is now open to all free users!