qdrant / quantization

Apache License 2.0
6 stars 4 forks source link

Use rayons fork join model in pq #12

Closed IvanPleshkov closed 1 year ago

IvanPleshkov commented 1 year ago

Parallel PQ data encoding using rayon crate.

rayon provides fork-join model https://docs.rs/rayon/latest/rayon/fn.scope.html.

While this PR, N threads for data processing is created. Each thread number thread_index process all vectors i that (i + thread_index) % max_threads == 0. Encoding result is protected by conditional variables that implements ordered access (see more in comments).