pisa-engine / pisa

PISA: Performant Indexes and Search for Academia
https://pisa-engine.github.io/pisa/book
Apache License 2.0
942 stars 65 forks source link

List Splitting/Clipping #565

Open JMMackenzie opened 10 months ago

JMMackenzie commented 10 months ago

Describe the solution you'd like There is a bunch of prior work on splitting postings lists; this allows the high impact list to be traversed (more likely) than the low impact list, and since the low impact lists are typically long, accelerates retrieval. Recent work in EMNLP findings showed that this is highly beneficial to learned sparse retrieval mechanisms. I would like to see if this can be pulled into the PISA engine with perhaps a more elegant design (I currently have a private prototype used in the EMNLP experiments but it's a bit rough for release still). This would then supersede the (currently empty) repo here: https://github.com/JMMackenzie/postings-clipping

Paper: https://aclanthology.org/2022.findings-emnlp.205/

Additional context TBA - This is a draft issue for now.