Add padding while saving ivf.pid.pt and release GIL

stanford-futuredata / ColBERT

ColBERT: state-of-the-art neural search (SIGIR'20, TACL'21, NeurIPS'21, NAACL'22, CIKM'22, ACL'23, EMNLP'23)

MIT License

2.67k stars 355 forks source link

Add padding while saving ivf.pid.pt and release GIL #336

Closed VThejas closed 1 month ago

VThejas commented 2 months ago

We append the padding before saving ivf.pid.pt. This is so that when we load this back, we can avoid the memory spike caused while appending the padding after loading. Functionally, there is no difference.

Also GIL is released for more parallelism.

VThejas commented 1 month ago

@santhnm2 @okhat I tested this out and it works, will merge this.