[+] Does this PR have a descriptive title that could go in our release notes?
[-] Does this PR add any new dependencies?
[-] Does this PR modify any existing APIs?
[-] Is the change to the API backwards compatible?
[+] Should this result in any changes to our documentation, either updating existing docs or adding new ones?
Added following functions:
generate_pq_pivots_simplified
generate_pq_data_from_pivots_simplified
New functions for product quantization that do not rely on retrieving data from files or storing any data in files.
These functions can be used by systems that rely on a different type of data storage.
Tests:
I instrumented existing functions for producing pivot data and generating pq data.
I could confirm that giving identical input (pivot data and input full vectors) a new function generate_pq_data_from_pivots_simplified() produces output identical to the original generate_pq_data_from_pivots() function.
Function generate_pq_pivots_simplified() produces output very similar to the original function generate_pq_pivots() given the same input. The differences can be explained by outputs of non-deterministic core functions for generating pivot data.
Added following functions: generate_pq_pivots_simplified generate_pq_data_from_pivots_simplified
New functions for product quantization that do not rely on retrieving data from files or storing any data in files. These functions can be used by systems that rely on a different type of data storage.
Tests: I instrumented existing functions for producing pivot data and generating pq data. I could confirm that giving identical input (pivot data and input full vectors) a new function generate_pq_data_from_pivots_simplified() produces output identical to the original generate_pq_data_from_pivots() function. Function generate_pq_pivots_simplified() produces output very similar to the original function generate_pq_pivots() given the same input. The differences can be explained by outputs of non-deterministic core functions for generating pivot data.