stanford-futuredata / ColBERT

ColBERT: state-of-the-art neural search (SIGIR'20, TACL'21, NeurIPS'21, NAACL'22, CIKM'22, ACL'23, EMNLP'23)
MIT License
2.67k stars 355 forks source link

How to set chunk_size #319

Open kevinningthu opened 3 months ago

kevinningthu commented 3 months ago

def get_chunksize(self): return min(25_000, 1 + len(self) // Run().nranks)

Seems like it is set to 25_000 when the dataset is very large.

How to set the parameter chunk_size?

Thanks!