Fixing the ability to resume index builds

Future-House / paper-qa

High accuracy RAG for answering questions from scientific documents with citations

Apache License 2.0

6.44k stars 618 forks source link

Fixing the ability to resume index builds #577

Closed jamesbraza closed 1 month ago

jamesbraza commented 1 month ago

https://github.com/Future-House/paper-qa/pull/513 moved us to just call save_index once at the end of an index build as an optimization. However, this had the unexpected side effect of breaking out ability to resume index builds. This regression took place because we had no testing of this. So, this PR:

Brings back save_index calls mid-indexing
Adds a test that index builds can be resumed