gingerwizard
commented
2 years ago Remaining work:
- [x] Field filtering on responses - we dont want the body in the responses
- [x] Implement optimize
- [x] Pagination (from and to)
- [x] Faceting e.g. on author
- [x] Filtering on facets
- [x] ~Separate fields on disk to speed things up. Possibly compress using lz4.~ Field reading is taking less than 10ms. Not worth doing this.
- [ ] catch exceptions - release any locks!
- [ ] Finish ui and style
Nice to have
- [ ] ~Lazy iteration doc ids and positions~ We dont need this given the size of the dataset. Only needed if memory becomes an issue.
- [ ] Autocomplete endpoint @lollobaldo
Performance improvements (if needed):
- [x] Postings are always loaded even if we dont them - we could store an offset for this but we need term, doc key (maybe in a large file). Maybe use faster encoding on disk e.g. quickle
- [x] ~Separate fields on disk i.e. body separately so we dont have to parse all of it just for the title~ - this isn't worth it. Its < 2ms to read the docs.
- [x] Skip lists for postings and positions
- [ ] For queries with more than 2 terms, we could evaluate the smallest pairs first - avoiding large intersections. Would speed up even more if combined with skip lists.
- [ ] Optimise pickle - dont copy state
- [x] Optimise reading across segments by assuming in order guarantees ~- [x] lz4 compression (might help reading speed)~ This doesn't offer an benefit
- [x] ~Delta encode positions < data size by 15% but didnt speed up queries - added read time~
- [ ] Could we continue indexing whilst flushing (would mean flushing would be pushed to another thread)
API will be a simple REST JSON service - in python as we'll need to integrate with the backend.
Frontend im proposing using https://inveniosoftware.github.io/react-searchkit/