CliDyn / climsight

prototype of a system that provide local climate information
BSD 3-Clause "New" or "Revised" License
29 stars 10 forks source link

RAG optimization: overwritten database #116

Open AntoniaJost opened 2 months ago

AntoniaJost commented 2 months ago

Right now, when one (or multiple) file(s) in data/ipcc_text_reports have changed, the entire chunking and embedding will take place again and get added on top of the already existing chunks, even if they already exist. Desired behaviour: only the file(s) that has/have changed or were newly added shall get chunked and embedded to save resources.