assertion always fail with GpuIndexIVFPQ

[hellolovetiger]

Here is the details of the error: Faiss assertion usePrecomputed_ || IVFPQ::isSupportedNoPrecomputedSubDimSize( this->d / subQuantizers_) failed in void faiss::gpu::GpuIndexIVFPQ::assertSettings_() const at GpuIndexIVFPQ.cu:469Aborted (core dumped)

And here is part of my code:

res = faiss.StandardGpuResources()

index = faiss.index_factory(d, "OPQ8,IVF4096,PQ8")

co = faiss.GpuClonerOptions()
co.useFloat16 = False
co.usePrecomputed = True
co.indicesOptions = faiss.INDICES_CPU

index = faiss.index_cpu_to_gpu(res, 0, index, co)  # error happends here

d is 1000.

[mdouze]

Hi

The ratio between the dimension d and the PQ size needs to be one of the values mentioned here:

https://rawgit.com/facebookresearch/faiss/master/docs/html/PQScanMultiPassNoPrecomputed_8cu_source.html

function isSupportedNoPrecomputedSubDimSize. The ratio here is 125, not supported.

To make it work, you can ask the OPQ pre-processing to reduce the # dimensions, with "OPQ8_512,IVF4096,PQ8". Another parameter is that it is often more efficient to keep the ratio between the PQ size and the dimension at arount 4-16, so if you choose a PQ8, it is probably best to just reduce the vectors to 128-D, ie. "OPQ8_128,IVF4096,PQ8".

I will add this to the Troubleshooting section of the wiki.

[hellolovetiger]

Thanks, mdouze. It works.