search

nmslib / hnswlib

Header-only C++/python library for fast approximate nearest neighbors
https://github.com/nmslib/hnswlib
Apache License 2.0
4.11k stars 607 forks source link

Question: Is index build time unstable? #548

Open wu3227834 opened 3 months ago

wu3227834 commented 3 months ago

Hello,

I'm testing the performance of HNSW (HNSWlib) on a 2c8g environment. I found that there is a difference in the time to construct the index under the same index parameters.

Data Set：Cohere 1M, 768 Dim Index Parameter：M:30,ef_construction:360、ef_search:100 Construction time(s)：4974.660、4570.640、5673.193、4870.791

Is HNSWlib index build time unstable?

Thank you!

yurymalkov commented 3 months ago

Hi @wu3227834,

It is likely that the node you are running is virtual and is shared across multiple users. So, the performance that you are measure depends on the programs that other users are running. If you get a full node, that should go away

wu3227834 commented 3 months ago

Hi @yurymalkov Thank u for ur reply I encapsulated hnswlib and used docker-compose to limit the 2c8g resource. I’m not sure if it’s a problem with my program. Theoretically, with the same index parameters, under the same environment resources and the same training set, the index construction time is stable, right?

Thank you!