search

vespa-engine / vespa

AI + Data, online. https://vespa.ai
https://vespa.ai
Apache License 2.0
5.84k stars 604 forks source link

add progress reporting when rebuilding HNSW index #32886

Closed arnej27959 closed 3 days ago

arnej27959 commented 3 days ago

@geirst please review @havardpe @andreer FYI

arnej27959 commented 3 days ago

example log snippet with this change:

[2024-11-19 13:51:25] EVENT   state/1 name="load.attribute.start" value="{"documentsubdb":"product.0.ready","name":"embedding"}"
[2024-11-19 13:51:27] WARNING Cannot use saved HNSW index for ANN, index parameters have changed
[2024-11-19 13:51:27] EVENT   state/1 name="hnsw.index.rebuild.start" value="{"name":"embedding","execution":"multi-threaded"}"
[2024-11-19 13:52:27] EVENT   state/1 name="hnsw.index.rebuild.progress" value="{"name":"embedding","percent":19.94368005631994}"
[2024-11-19 13:53:27] EVENT   state/1 name="hnsw.index.rebuild.progress" value="{"name":"embedding","percent":35.53376446623553}"
[2024-11-19 13:54:27] EVENT   state/1 name="hnsw.index.rebuild.progress" value="{"name":"embedding","percent":50.36084963915036}"
[2024-11-19 13:55:27] EVENT   state/1 name="hnsw.index.rebuild.progress" value="{"name":"embedding","percent":64.71463528536472}"
[2024-11-19 13:56:27] EVENT   state/1 name="hnsw.index.rebuild.progress" value="{"name":"embedding","percent":78.77072122927878}"
[2024-11-19 13:57:27] EVENT   state/1 name="hnsw.index.rebuild.progress" value="{"name":"embedding","percent":92.42170757829243}"
[2024-11-19 13:58:00] EVENT   state/1 name="hnsw.index.rebuild.complete" value="{"name":"embedding","time.elapsed.ms":393512.0}"
[2024-11-19 13:58:00] EVENT   state/1 name="load.attribute.complete" value="{"documentsubdb":"product.0.ready","name":"embedding","time.elapsed.ms":395209}"