Created an index via diskannpy.build_memory_index(), then searched it via diskannpy.StaticMemoryIndex.search()
# points is a np.ndarray of dtype np.float32, ~ 16m points x 1024 dims
dap.build_memory_index(
data=points,
distance_metric="cosine",
index_directory="/home/daxpryce/data/index",
complexity=128,
graph_degree=100,
num_threads=0
)
...
index = dap.StaticMemoryIndex(
index_directory="/home/daxpryce/data/index",
initial_search_complexity=128,
num_threads=0
)
...
index.search(query_vector, k_neighbors=25, complexity=128)
I expect this to give me the 25 ANN identifiers and their respective distances, with 0 being the closest and 1 or 2 being the most different (note: I don't actually know if it should be 1 or 2 here)
Actual Behavior
I get back the nearest neighbors in the identifiers part of the NamedTuple, but distances are coming back as:
Please tell us about the shape and datatype of your data, (e.g. 128 dimensions, 12.3 billion points, floats)
Dimensions: 1024
Number of Points: 16_287_459
Data type: float32
Your Environment
Operating system (e.g. Windows 11 Pro, Ubuntu 22.04.1 LTS): Ubuntu 20.04.6 LTS
DiskANN version (or commit built from): 0.7.0 (from pypi)
Additional Details
The query vectors aren't being normalized prior to searching. The relative results seem to be correct, but if I manually normalize the query vector and then search, I get distances within ranges I would fully expect. Using scipy.spatial.distance.cdist(..., ..., metric="cosine") I verified that these distances are not, in fact, -27. I also verified that if I manually normalize the query vector prior to search, I do get identical distances.
Expected Behavior
Created an index via
diskannpy.build_memory_index()
, then searched it viadiskannpy.StaticMemoryIndex.search()
I expect this to give me the 25 ANN identifiers and their respective distances, with 0 being the closest and 1 or 2 being the most different (note: I don't actually know if it should be 1 or 2 here)
Actual Behavior
I get back the nearest neighbors in the
identifiers
part of the NamedTuple, butdistances
are coming back as:record scratch -27.86? come again? cosine? no.
Dataset Description
Please tell us about the shape and datatype of your data, (e.g. 128 dimensions, 12.3 billion points, floats)
Your Environment
Additional Details
The query vectors aren't being normalized prior to searching. The relative results seem to be correct, but if I manually normalize the query vector and then search, I get distances within ranges I would fully expect. Using scipy.spatial.distance.cdist(..., ..., metric="cosine") I verified that these distances are not, in fact, -27. I also verified that if I manually normalize the query vector prior to search, I do get identical distances.
Long story short, search does not always normalize the query vector. I also verified this happens on the native diskann class object created via https://github.com/microsoft/DiskANN/blob/9e876376349132236723d1d167c071887d807e9c/python/src/static_memory_index.cpp#L34
Results:
When I normalize it:
Results:
non-diskannpy/diskann cosine distance calcs:
Results: Note: I didn't bother with vector @
5345615
, as that was the query vector.Last note: I am like 99% confident I have seen this work properly.