search

multimodal-art-projection / MAP-NEO

877 stars 81 forks source link

How much memory is needed when doing MinHashLSH dedup? #11

Closed SefaZeng closed 4 months ago

SefaZeng commented 6 months ago

Thanks.

panDing19 commented 4 months ago

It depends on the number of document in your dataset. In our setting, 256 gb * 80 is already enought.