Ingesting your own dataset

[bouiboui]

Hi, when ingesting my own dataset, I understand that I must remove/replace Export*.zip and Notion_DB. Should I also remove faiss_store.pkl? (I've never used faiss) Are there other things that need to be removed/reset first? Thanks!

[jasielmacedo]

According to my understand of this code and tests that I did, the Export*.zip seems to be a leftover because all the content lives in the Notion_DB folder. If you replace the Notion_DB folder putting your own data (keeping the folder name of course), just execute ingest.py providing the openAI key and faiss will auto update the faiss_store.pkl

[bouiboui]

But does it mean it will also have Blendle's employee handbook information?

[jasielmacedo]

Yes and no, because the vectors could be there (locally) but in order to use them, you have to call the Open AI Embedding API to generate those vectors on Open AI side as well. But I recommend delete faiss_store.pkl, the Faiss library will generate a new one anyway

[bouiboui]

Alright, thanks !