search

facebookresearch / faiss

A library for efficient similarity search and clustering of dense vectors.
https://faiss.ai
MIT License
30.72k stars 3.58k forks source link

Fix the endianness issue in AIX while running the benchmark. #3330

Open KamathForAIX opened 6 months ago

KamathForAIX commented 6 months ago

Summary

When we run the IVF_FASTSCAN benchmark on AIX, we are getting lesser recall values compared to other destros.

For example for IVF_FASTSCAN we get: On AIX: ======IVF1024,PQ32x4fs nprobe 1, Recall@1: 0.041200, speed: 0.048774 ms/query nprobe 2, Recall@1: 0.039400, speed: 0.090845 ms/query nprobe 4, Recall@1: 0.037600, speed: 0.175046 ms/query nprobe 6, Recall@1: 0.036500, speed: 0.255538 ms/query nprobe 8, Recall@1: 0.035000, speed: 0.336469 ms/query

on Linux: ======IVF1024,PQ32x4fs nprobe 1, Recall@1: 0.922900, speed: 0.064736 ms/query nprobe 2, Recall@1: 0.943900, speed: 0.122122 ms/query nprobe 4, Recall@1: 0.955800, speed: 0.237951 ms/query nprobe 6, Recall@1: 0.958000, speed: 0.350556 ms/query nprobe 8, Recall@1: 0.960000, speed: 0.461979 ms/query.

AIX uses a big-endian system. When we code data into uint8 arrays and then cast them into uint16, the permutation arrays are swapped in an in an order different from the way AIX OS expects them to be, like in the search functions.

Our proposed fix is to modify the array const uint8_t perm0[16] = 0, 8, 1, 9, 2, 10, 3, 11, 4, 12, 5, 13, 6, 14, 7, 15} in impl/pq4_fast_scan.cpp to big endian in case the system is a big endian system. This file's functions pack codes into uint8 arrays. These packed code arrays are cast to uint16 in the search routines found in impl/pq4_fast_scan_search_1.cpp and impl/pq4_fast_scan_search_qbs.cpp. As a result, the code ordering changes across Linux and AIX because of endianness differences. It switches the data in the packed code arrays for AIX to how the search functions want it to by altering the permutation arrays.

In order to fix the same, we would like to seek your guidance as well as propose a fix to this issue. Kindly let us know what you think.

Platform

OS: AIX

Faiss version: Master branch

Installed from: Compiled myself

Faiss compilation options: export CC="/opt/IBM/openxlC/17.1.2/bin/ibm-clang " export CXX="/opt/IBM/openxlC/17.1.2/bin/ibm-clang++_r " export CXXFLAGS="-m64 -fopenmp -I/usr/local/include -I/opt/freeware/include" export CFLAGS="-m64 -fopenmp -I/usr/local/include -I/opt/freeware/include" export LDFLAGS="-bbigtoc"

Reproduction instructions

Run the benchmark in AIX in the benchs directory. In our case, it is python3 bench_ivf_fastscan.py

KamathForAIX commented 3 months ago

@mdouze Any estimate on when we can expect the new release of FAISS?

Kindly let us know.