Improve Portal Documentation (Data Catalog, How it Works, ...)

davidgasquez / gitcoin-grants-data-portal

🌲 Open source, serverless, and local-first data hub for Gitcoin Grants data!

https://grantsdataportal.xyz/

MIT License

26 stars 3 forks source link

Improve Portal Documentation (Data Catalog, How it Works, ...) #29

Open davidgasquez opened 8 months ago

davidgasquez commented 8 months ago

We should expose both schemas and samples for all the curated datasets.

This will improve UX and make choosing datasets easier!

davidgasquez commented 8 months ago

@DistributedDoge mentioned to publish a notebook that loops over tables or whatever export dir is, fetches schema from .parquetfiles and surfaces that.

DistributedDoge commented 8 months ago

The code has landed, catalog will be built next time you update github pages by doing make publish.

Just remember to do make run first so that notebook can access table schema from which catalog is being built.

davidgasquez commented 8 months ago

The code has landed, catalog will be built next time you update github pages by doing make publish. :facepalm:

I thought the website was published with each push! :shrug: Got confussed with the Filecoin one.

Created an issue now: https://github.com/davidgasquez/gitcoin-grants-data-portal/issues/36

davidgasquez commented 8 months ago

Working! :tada:

https://davidgasquez.github.io/gitcoin-grants-data-portal/catalog.html

DistributedDoge commented 8 months ago

Pretty neat. Three things I will try to add later:

[ ] download link to each file, made easier thanks to stable links in #18
[ ] file size + row count so folks know how much data they have on hand
[ ] better display for columns with nested schema

davidgasquez commented 8 months ago

Sharing it here so I remember in the future.

Would be awesome to aim for something like this: https://py-code.org/datasets

Nice UX and UI!

DistributedDoge commented 7 months ago

Also, on catalog side,

[ ] Detailed descriptions for 3 most used tables (rounds, projects, rounds_votes)
[ ] Introductory paragraph (data is mainly from indexer, updated at least weekly etc.)

Inspiration:

https://docs.passport.gitcoin.co/building-with-passport/passport-api/data-dictionary

davidgasquez commented 7 months ago

Cool find! I think we can do something similar with Dagster assets metadata.

Similar to what Subsets does.

Not sure how to deal with dbt models though! Perhaps we can extract the docs from the YAML files or even better, make Dagster understand dbt docs. :thinking: