Open davidgasquez opened 8 months ago
@DistributedDoge mentioned to publish a notebook that loops over tables
or whatever export dir is, fetches schema from .parquet
files and surfaces that.
The code has landed, catalog will be built next time you update github pages by doing make publish
.
Just remember to do make run
first so that notebook can access table schema from which catalog is being built.
The code has landed, catalog will be built next time you update github pages by doing make publish. :facepalm:
I thought the website was published with each push! :shrug: Got confussed with the Filecoin one.
Created an issue now: https://github.com/davidgasquez/gitcoin-grants-data-portal/issues/36
Pretty neat. Three things I will try to add later:
file size + row count
so folks know how much data they have on handSharing it here so I remember in the future.
Would be awesome to aim for something like this: https://py-code.org/datasets
Nice UX and UI!
Also, on catalog side,
Inspiration:
https://docs.passport.gitcoin.co/building-with-passport/passport-api/data-dictionary
Cool find! I think we can do something similar with Dagster assets metadata
.
Similar to what Subsets does.
Not sure how to deal with dbt models though! Perhaps we can extract the docs from the YAML files or even better, make Dagster understand dbt
docs. :thinking:
We should expose both schemas and samples for all the curated datasets.
This will improve UX and make choosing datasets easier!