search

AI-Northstar-Tech / vector-io

Comprehensive Vector Data Tooling. The universal interface for all vector database, datasets and RAG platforms. Easily export, import, backup, re-embed (using any model) or access your vector data from any vector databases or repository.
https://vector-io.com
Apache License 2.0
222 stars 27 forks source link

Refactoring removed critical / necessary code #89

Open tkreuder opened 7 months ago

tkreuder commented 7 months ago

Details

https://github.com/AI-Northstar-Tech/vector-io/commit/23a217dd5c8717536260bef842064222621fcb15 removed code, so that pinecone export does not work anymore

Branch

No response

greptile-apps[bot] commented 7 months ago

To address the issue with Pinecone export functionality not working due to code removal in the commit 23a217dd5c8717536260bef842064222621fcb15, please review the current implementation of the ExportPinecone class in src/vdf_io/export_vdf/pinecone_export.py. This file contains the necessary methods for exporting data from Pinecone, including initialization, data fetching, and saving to a parquet file. If the removed code is critical for the operation, consider reverting the changes or re-implementing the missing functionality based on the current structure of the ExportPinecone class and its methods. Additionally, examining the src/vdf_io/export_vdf directory may provide insights into how export functionalities are structured across different databases, which could be helpful in restoring or enhancing the Pinecone export feature.

References

ai-northstar-tech/vector-io/src/vdf_io/export_vdf/pinecone_export.py ai-northstar-tech/vector-io/src/vdf_io/export_vdf

Ask Greptile

dhruv-anand-aintech commented 7 months ago

Hi @tkreuder, this commit was part of a PR https://github.com/AI-Northstar-Tech/vector-io/pull/86 which added back the necessary code under a flag. If the new list() function fails to list the ids in the index, it uses the old random search method to get the points. Can you share the command you use to export, the verson of vecto-io you used, and the error you got?

dhruv-anand-aintech commented 7 months ago

I'm looking into the issue further and it looks like the function update_range_from_new_ids didn't get added back. testing a fix now

dhruv-anand-aintech commented 7 months ago

can you retry with 0.1.246?

And if it doesn't work, try again using the flag --no-use_list_points after pinecone in your command

dhruv-anand-aintech commented 6 months ago

@tkreuder could you confirm if this fixed it for you?