Uneccessary reprocessing with datalake storage gen2

microsoft / AI-Chat-App-Hack

HackTogether: The Microsoft Python Chatbot Hack | Register, Hack, Win

MIT License

234 stars 49 forks source link

Uneccessary reprocessing with datalake storage gen2 #147

Closed fragpk closed 4 months ago

fragpk commented 6 months ago

With datalake storage, every time i run azd up, it processes docs from the beggining. This wasn't happening with local files. How can we avoid the uneccessary reprocessing?

pamelafox commented 4 months ago

Could you file this issue in https://github.com/Azure-Samples/azure-search-openai-demo/ if you haven't already? I think we have some improvements to make with ADLS2 ingestion.