Use HFFileSystem to allow import to VDB directly from HD Dataset (without cloning)

AI-Northstar-Tech / vector-io

Comprehensive Vector Data Tooling. The universal interface for all vector database, datasets and RAG platforms. Easily export, import, backup, re-embed (using any model) or access your vector data from any vector databases or repository.

https://vector-io.com

Apache License 2.0

220 stars 27 forks source link

Use HFFileSystem to allow import to VDB directly from HD Dataset (without cloning) #31

Closed dhruv-anand-aintech closed 9 months ago

dhruv-anand-aintech commented 9 months ago

https://huggingface.co/docs/huggingface_hub/package_reference/hf_file_system

dhruv-anand-aintech commented 9 months ago

supported via hf_dataset cli argument now