microsoft / DiskANN

Graph-structured Indices for Scalable, Fast, Fresh and Filtered Approximate Nearest Neighbor Search
Other
1.02k stars 208 forks source link

[Question] Parallel index building strategy #540

Open kisasexypantera94 opened 4 months ago

kisasexypantera94 commented 4 months ago

Hi, I see that you are parallelizing index construction using per-point locks and I’m wondering if maybe you’ve tried other methods? Like run kmeans and build these clusters in parallel, and then merge the edges. Or maybe take immutable snapshots of the current state of the index and run point updates in parallel on this state, repeating the process several iterations