Currently the id for each vector entry in pinecone is a randomly generated ID. TO prevent duplicates in the destination index, a step was introduces to persist the record id from the FileData object in the metadata and delete using that during upload to get rid of duplicates.
Description
Currently the id for each vector entry in pinecone is a randomly generated ID. TO prevent duplicates in the destination index, a step was introduces to persist the record id from the
FileDataobject in the metadata and delete using that during upload to get rid of duplicates.