alan-turing-institute / reginald

Reginald repository for REG Hack Week 23
3 stars 0 forks source link

Updating data indexes #124

Open rchan26 opened 11 months ago

rchan26 commented 11 months ago

Whenever we want to update our data indexes (perhaps there's new data / updates in the repos that we pull from), we need to rebuild the full index which can be expensive and time-consuming. It would be good to figure out what's been updated and what to add.

We have done this before in this notebook manually, but note this is loading in data from files rather than GitHub readers which we use now.

rchan26 commented 11 months ago

See https://docs.llamaindex.ai/en/stable/core_modules/data_modules/index/document_management.html#update