search

aorwall / moatless-tools

MIT License
216 stars 20 forks source link

Making EpicSplitter persistent? #9

Open zkx06111 opened 2 months ago

zkx06111 commented 2 months ago

I really appreciate the notebook for efficiently building the index by using the previously persisted index.

However, this can still be slow as the splitting process still needs to run from scratch for every instance (correct me if I'm wrong).

Do you think the splitting process can also be incremental and utilizing previously split data? I think tree-sitter is incremental in parsing the AST.

aorwall commented 2 months ago

Yes, would make a lot of sense. Right now all files are also parsed/split every time even if they're not changed since the last commit also. So both these changes would be nice to improve performance.

For the embedding part of the ingestion I've also been thinking about ways to detect if there are significant changes and only embed chunks again if there are any. Haven't figured out how though...