asg017 / sqlite-vss

A SQLite extension for efficient vector search, based on Faiss!
MIT License
1.75k stars 62 forks source link

Python - sqlite_vss.load(conn) fail without any error message #105

Open yandod opened 1 year ago

yandod commented 1 year ago

I am trying python bindings on Docker on M1 mac with amd64 image. If anyone can hint next step for debugging, I would greatly appreciate it.

I do suspect side effect of emulation for now.

print(sqlite_vss.vss_loadable_path()) print('initilizing SQLite...') conn = sqlite3.connect('test.db') conn.enable_load_extension(True) print('initilizing VSS...') sqlite_vss.load(conn)

conn.load_extension(sqlite_vss.vss_loadable_path())

print(conn.execute('select vss_version()').fetchone()[0])


- output
```console
python3 /app/debug.py
/usr/local/lib/python3.7/site-packages/sqlite_vss/vss0
initilizing SQLite...
initilizing VSS...

RUN apt-get update && \ apt-get install -y --no-install-recommends \ libgomp1 \ libatlas-base-dev \ liblapack-dev && \ apt-get clean && \ rm -rf /var/lib/apt/lists/*

COPY ./app /app WORKDIR /app

RUN pip install -r requirements.txt

yandod commented 1 year ago

tried another shot with disabling "Use Rosetta for emulation" and got this output.

/usr/local/lib/python3.7/site-packages/sqlite_vss/vss0
initilizing SQLite...
initilizing VSS...
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
qemu: uncaught target signal 6 (Aborted) - core dumped
yandod commented 1 year ago

checked installed packages bellow

 dpkg -s libgomp1 libatlas-base-dev liblapack-dev      
Package: libgomp1
Status: install ok installed
Priority: optional
Section: libs
Installed-Size: 312
Maintainer: Debian GCC Maintainers <debian-gcc@lists.debian.org>
Architecture: amd64
Multi-Arch: same
Source: gcc-12
Version: 12.2.0-14
Depends: gcc-12-base (= 12.2.0-14), libc6 (>= 2.34)
Breaks: gcc-4.3 (<< 4.3.6-1), gcc-4.4 (<< 4.4.6-4), gcc-4.5 (<< 4.5.3-2)
Description: GCC OpenMP (GOMP) support library
 GOMP is an implementation of OpenMP for the C, C++, and Fortran compilers
 in the GNU Compiler Collection.
Homepage: http://gcc.gnu.org/

Package: libatlas-base-dev
Status: install ok installed
Priority: optional
Section: libdevel
Installed-Size: 23396
Maintainer: Debian Science Team <debian-science-maintainers@lists.alioth.debian.org>
Architecture: amd64
Multi-Arch: same
Source: atlas
Version: 3.10.3-13
Replaces: libatlas-dev (<< 3.10.3-2), libcblas-dev
Provides: libblas.so, liblapack.so
Depends: libatlas3-base (= 3.10.3-13)
Suggests: libatlas-doc, liblapack-doc
Breaks: libatlas-dev (<< 3.10.3-2), libblas-dev (<< 3.7.1-2~), liblapack-dev (<< 3.7.1-2~), libopenblas-dev (<< 0.2.20+ds-3~)
Conflicts: libcblas-dev
Description: Automatically Tuned Linear Algebra Software, generic static
 ATLAS is an approach for the automatic generation and optimization of
 numerical software. Currently ATLAS supplies optimized versions for the
 complete set of linear algebra kernels known as the Basic Linear Algebra
 Subroutines (BLAS), and a subset of the linear algebra routines in the
 LAPACK library.
 .
 This package includes the headers, the static libraries and symbolic links
 needed for program development.
Homepage: http://math-atlas.sourceforge.net/

Package: liblapack-dev
Status: install ok installed
Priority: optional
Section: libdevel
Installed-Size: 25055
Maintainer: Debian Science Team <debian-science-maintainers@lists.alioth.debian.org>
Architecture: amd64
Multi-Arch: same
Source: lapack
Version: 3.11.0-2
Provides: liblapack.so
Depends: liblapack3 (= 3.11.0-2), libblas-dev | libblas.so
Suggests: liblapack-doc
Description: Library of linear algebra routines 3 - static version
 LAPACK version 3.X is a comprehensive FORTRAN library that does
 linear algebra operations including matrix inversions, least
 squared solutions to linear sets of equations, eigenvector
 analysis, singular value decomposition, etc. It is a very
 comprehensive and reputable package that has found extensive
 use in the scientific community.
 .
 This package contains a static version of the library.