faiss::Clustering objective remains zero when training on GPU

[kisasexypantera94]

Summary

Platform

OS: Docker container nvidia/cuda:11.6.2-devel-ubuntu20.04 Nvidia: GTX 1060, Driver Version: 515.76 CUDA Version: 11.7

Faiss version: 1.7.2

Installed from: manual build with CMake

Faiss compilation options: none

Running on:

• [ ] CPU
• [x] GPU

Interface:

• [x] C++
• [ ] Python

Reproduction instructions

Sample code from wiki:

#include <cstdio>
#include <vector>

#include <faiss/Clustering.h>
#include <faiss/utils/utils.h>
#include <faiss/utils/random.h>
#include <faiss/gpu/GpuIndexFlat.h>
#include <faiss/gpu/StandardGpuResources.h>

std::vector<float> makeRandomVecs(size_t numVecs, int dim) {
  std::vector<float> vecs(numVecs * dim);
  faiss::float_rand(vecs.data(), vecs.size(), 1);
  return vecs;
}

int main(int argc, char** argv) {
  faiss::gpu::StandardGpuResources res;

  int dim = 128;
  int numberOfEMIterations = 20;
  size_t numberOfClusters = 2000;
  size_t numVecsToCluster = 500000;

  std::vector<float> vecs = makeRandomVecs(numVecsToCluster, dim);
  faiss::gpu::GpuIndexFlatConfig config;
  config.device = 0;            // this is the default
  config.useFloat16 = false;    // this is the default
  faiss::gpu::GpuIndexFlatL2 index(&res, dim, config);

  faiss::ClusteringParameters cp;
  cp.niter = numberOfEMIterations;
  cp.verbose = true; // print out per-iteration stats

  faiss::Clustering kMeans(dim, numberOfClusters, cp);

  kMeans.train(numVecsToCluster, vecs.data(), index);

  printf("centroid 3 dim 6 is %f\n", kMeans.centroids[3 * dim + 6]);

  return 0;
}

produces that:

Clustering 500000 points in 128D to 2000 clusters, redo 1 times, 20 iterations
  Preprocessing in 0.18 s
  Iteration 19 (10.91 s, search 3.14 s): objective=0 imbalance=704.986 nsplit=1992       
centroid 3 dim 6 is 0.487962

as you can see objective is zero.

I've tried changing seed but that didn't help. CPU version works fine though. Also tried doing the same thing with Python API and it works fine too (faiss-gpu 1.7.2).

[kisasexypantera94]

Checked the same code on RTX 3050 and it works

[dvgrinberg]

Also noticed the same behaviour when linking to Intel MKL and running on CPU

[mdouze]

So the error is GPU specific and specific to the GTX 1060. Correct?

[kisasexypantera94]

So the error is GPU specific and specific to the GTX 1060. Correct?

Yes, tested on RTX 2080/3050 and they were working fine.

Thought the problem was with me not specifying cuda architecture, so I rebuilt faiss:

RUN pip3 install numpy
RUN apt install -y libopenblas-dev
RUN wget https://github.com/facebookresearch/faiss/archive/refs/tags/v1.7.2.tar.gz && \
    tar xf v1.7.2.tar.gz && \
    cd faiss-1.7.2 && mkdir build && cd build && \
    cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES="61;75;86" -DBUILD_TESTING=0 -DFAISS_OPT_LEVEL=generic -DFAISS_ENABLE_PYTHON=OFF -DFAISS_ENABLE_GPU=ON .. && \
    make -j10 faiss && make install

but still the same result