ETL support for Parquet - Githubissues

nickna / Neighborly

An open-source vector database

MIT License

9 stars 2 forks source link

ETL support for Parquet #44

Closed nickna closed 2 weeks ago

nickna commented 2 weeks ago

Description:

Add functionality to import and export vector data in Parquet format.

Tasks:

Implement data import functionality for Parquet format.
Implement data export functionality for the same format.
Write documentation and examples demonstrating how to use the import/export features.

Resources:

Parquet in .NET

Impact:

This functionality will make it easier for users to integrate Neighborly with their existing data workflows, enhancing its usability and versatility.

How to Contribute:

Fork the repository and create a new branch for your changes.
Implement the import/export functionality and integrate it into the framework.
Ensure all new and existing tests pass.
Submit a pull request with a detailed description of the changes.

Related to Issue #25

nickna commented 2 weeks ago

Parquet import is partially implemented. It's not working. It's up for grabs if anyone else wants to pick it up. If not, I'll revisit it after going through bug fixes.

I was testing it against this vectorized Wikipedia repo. It's a hard but very valid test for real-world data.