sgt1796 / GPT_embedding

0 stars 1 forks source link

Automation of embedding enrichment #7

Open sgt1796 opened 3 weeks ago

sgt1796 commented 3 weeks ago

need a way to automatically integrating newly collect data into existing database Fairly simple on raw data after embedding -- just add new rows at the end of the file

difficulty: IVFPQ index requires clustering before building, sample to cluster should be as evenly distributed as possible. Need to consider the impact of adding new data to database.