test: Streaming vector search benchmark - Githubissues

tensorchord / VectorChord

Scalable, Fast, and Disk-friendly Vector search in Postgres

GNU Affero General Public License v3.0

11 stars 2 forks source link

test: Streaming vector search benchmark #37

Open VoVAllen opened 2 weeks ago

VoVAllen commented 2 weeks ago

Reference: https://github.com/harsha-simhadri/big-ann-benchmarks/tree/main/neurips23/streaming

kemingy commented 5 days ago

code: https://github.com/harsha-simhadri/big-ann-benchmarks/pull/318

kemingy commented 3 days ago

test on AWS i4i.large with 2vCPU + 16G RAM + NVME
dataset: GIST 1M
max workers: set to vCPU number
steps
- insert 700k
- build index
- insert 300k
- query (single thread, after warmup)
config
- (pgvector 0.8.0) hnsw: top10 ef_search = 384 top100 ef_search = 512
- (vchord 0.1.0) vchordrq: lists = 4 * int(sampling ** 0.5) = 3344, external lists = 4096, top10 probe = 150 top100 probe = 200

Index	Build Index (seconds)	Insert After Build (seconds)	Query QPS/Recall (top10)	Query QPS/Recall (top100)
HNSW (pgvector)	3043.39	2052.36	48.59 / 0.9590	36.44 / 0.9373
RaBitQ (vchord)	2804.59	152.82	195.77 / 0.9675	111.57 / 0.9623
RaBitQ (vchord-external)	186.25	144.89	226.22 / 0.9588	122.83 / 0.9517