search

davidgasquez / gitcoin-grants-data-portal

🌲 Open source, serverless, and local-first data hub for Gitcoin Grants data!
https://grantsdataportal.xyz/
MIT License
26 stars 3 forks source link

Publish Parquet files under an static endpoint #5

Closed davidgasquez closed 8 months ago

davidgasquez commented 8 months ago

We're putting Parquet files on IPFS. Every release will be under different a IPFS hash and that is both confusing and hard to navigate from the UX side.

Will aim to rely on Filebase and offer an static IPNS endpoint for the exported Parquet files.

DistributedDoge commented 8 months ago

Having static endpoint for latest data is a great idea! Just two notes here:

1) When trying to download any of files published using Filebase gateway which is linked on Github Pages link I am being served This gateway only serves content pinned by Filebase string instead of file contents.

Simple workaround for me is to fetch the same content from another gateway like cloudflare-gateway but I am just curious why Filebase does not seem to recognize they are the ones pinning.

  1. In previous version I really enjoyed simplicitly of being able to download single .duckdb file with pre-loaded tables. Now I am able to cherry-pick tables and load those myself, but a way to get everything in single fetch with batteries included would be nice.
davidgasquez commented 8 months ago

but I am just curious why Filebase does not seem to recognize they are the ones pinning

Sounds like a bug! IPNS is tricky. Will reach out to Filebase and let them know. Meanwhile, we can rely on the IPFS CID file.

In previous version I really enjoyed simplicitly of being able to download single .duckdb file with pre-loaded tables. Now I am able to cherry-pick tables and load those myself, but a way to get everything in single fetch with batteries included would be nice.

You're right! In fact, it doesn't cost much to also push the duckdb database to GitHub releases as I was doing. Might be helpful for (1) people that want one file with everything and (2) as a backup in case IPFS is slow or the CID breaks for some reason.

davidgasquez commented 8 months ago

Closing this one as the Parquet files are now being pushed to IPFS.

Will work on https://github.com/davidgasquez/gitcoin-grants-data-portal/issues/8.