Datasets: Adding doc for DuckDB CLI integration

huggingface / hub-docs

Docs of the Hugging Face Hub

http://hf.co/docs/hub

Apache License 2.0

255 stars 222 forks source link

Datasets: Adding doc for DuckDB CLI integration #1297

Closed AndreaFrancis closed 2 months ago

AndreaFrancis commented 2 months ago

Following up https://github.com/duckdb/duckdb/pull/11831 for DuckDB CLI integration:

Authentication for private and gated datasets (Using DuckDB Secrets Manager)
Query datasets (Some basic SELECT examples, DESCRIBE, SUMMARIZE for stats)
Perform SQL operations (Text functions and aggregations)
Combine datasets, export and publish on the Hub
Perform vector similarity search

HuggingFaceDocBuilderDev commented 2 months ago

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

AndreaFrancis commented 2 months ago

I am not sure why the sections for DuckDB don't seem to be working as expected in https://moon-ci-docs.huggingface.co/docs/hub/pr_1297/en/datasets-duckdb

lhoestq commented 2 months ago

I think the tabs on the left are buggy in the preview but work when deployed. The other pages at https://moon-ci-docs.huggingface.co/docs/hub/pr_1297/en/datasets-duckdb are working though.

lhoestq commented 2 months ago

The current section main page is https://moon-ci-docs.huggingface.co/docs/hub/pr_1297/en/datasets-duckdb and shows the old way of using duckdb (register the filesystem), maybe we can set datasets-duckdb-cli as the main section page instead, and add links to the other pages at the end of this file ?

AndreaFrancis commented 2 months ago

The current section main page is https://moon-ci-docs.huggingface.co/docs/hub/pr_1297/en/datasets-duckdb and shows the old way of using duckdb (register the filesystem), maybe we can set datasets-duckdb-cli as the main section page instead, and add links to the other pages at the end of this file ?

Done, I moved to datasets-duckdb-cli instead to be the main section and added the first part of the old file (about what is DuckDB). I didn't copy the rest because it is already covered (how to upload the results to a new dataset repository).

AndreaFrancis commented 2 months ago

Thanks for all your feedback, @severo @lhoestq and @polinaeterna; I think it is ready and will merge.

severo commented 2 months ago

live here: https://huggingface.co/docs/hub/datasets-duckdb