microsoft / DiskANN

Graph-structured Indices for Scalable, Fast, Fresh and Filtered Approximate Nearest Neighbor Search
Other
1.02k stars 208 forks source link

[BUG] build_disk_index does not respect memory limit #553

Open gopalrs opened 3 months ago

gopalrs commented 3 months ago

Expected Behavior

build_disk_index should not exceed the memory limit set in the command line.

Actual Behavior

It consumes all available memory instead.

Dataset Description

All datasets

Error

Runs out of memory occasionally

Your Environment

Additional Details

Most likely it is because tcmalloc isn't releasing memory freed by DiskANN back to the OS.