microsoft / DiskANN

Graph-structured Indices for Scalable, Fast, Fresh and Filtered Approximate Nearest Neighbor Search
Other
1.16k stars 227 forks source link

[Question] How can I reduce memory cost when loading index & searching #566

Open hanqiushi opened 4 months ago

hanqiushi commented 4 months ago

When I run search_memory_index & search_disk_index in https://github.com/microsoft/DiskANN/blob/main/workflows/SSD_index.md , I found the memory cost is no less than other kinds of indexes. Is there any way I can run load index & search with less memory cost? Thanks

e.g. in search_disk_index, use pq_flash_index.cached_beam_search to search, after test case completed, the memory cost raises about 50MB, which is almost the file size of sift_learn.fbin