Pre-compiled loadable extension won't load on Python (3.10.8, Win11)

10c8 commented 4 months ago

Trying to load the extension manually like so:

conn.enable_load_extension(True)
conn.load_extension("vec0.dll")
conn.enable_load_extension(False)

Results in the following error:

OperationalError: Não foi possível encontrar o módulo especificado.

Which roughly translates to "The specified module could not be found". Specifying the full path to the file does not work either.

It could have to do with the sqlite3 DLL that comes with Python and the extension DLL being built using different compilers. I tried builting the extension myself with MSVC, using the following command:

cl -W -I.\sqlite-autoconf sqlite-vec.c sqlite-vec.h -O2

But this fails with lots of syntax ("initializer is not const", etc.) and linking errors.

asg017 commented 4 months ago

Hey thanks for the report - are you on 64-bit or 32-bit version of windows? Can you also share the Python version + SQLite version your python install has? a print(sqlite3.sqlite_version) should work there

Also when compiling yourself, try compiling it into a dynamic library with -fPIC -shared flags, and -o vec0.dll. Curious to see if that works, like:

cl -W -I.\sqlite-autoconf sqlite-vec.c sqlite-vec.h -O2 -fPIC -shared -o vec0.dll

Could you also try pip install sqlite-vec and this Python code:

import sqlite3
import sqlite_vec

db = sqlite3.connect(":memory:")
db.enable_load_extension(True)
sqlite_vec.load(db)
db.enable_load_extension(False)
print(db.execute("select vec_version()").fetchone())

Adriankhl commented 4 months ago

Python loading doesn't work, sqlite 3.45.1

In [1]: import sqlite3
   ...: import sqlite_vec

In [2]: db = sqlite3.connect(":memory:")
   ...: db.enable_load_extension(True)
   ...: sqlite_vec.load(db)
   ...: db.enable_load_extension(False)
   ...: print(db.execute("select vec_version()").fetchone())
---------------------------------------------------------------------------
OperationalError                          Traceback (most recent call last)
Cell In[2], line 3
      1 db = sqlite3.connect(":memory:")
      2 db.enable_load_extension(True)
----> 3 sqlite_vec.load(db)
      4 db.enable_load_extension(False)
      5 print(db.execute("select vec_version()").fetchone())

File ~\scoop\apps\python311\current\Lib\site-packages\sqlite_vec\__init__.py:13, in load(conn)
     12 def load(conn: sqlite3.Connection)  -> None:
---> 13   conn.load_extension(loadable_path())

OperationalError: The specified module could not be found.

Adriankhl commented 4 months ago

I guess you use mingw to compile for windows, because clang-cl works

clang-cl.exe /D_USRDLL /D_WINDLL sqlite-vec.c /MT /link /DLL /OUT:sqlite-vec.dll

but cl doesn't

cl.exe /D_USRDLL /D_WINDLL sqlite-vec.c /MT /link /DLL /OUT:sqlite-vec.dll

So it is probably a gcc extension problem

Error message:

sqlite-vec.c(378): error C2036: 'void *': unknown size
sqlite-vec.c(2362): error C2036: 'void *': unknown size
sqlite-vec.c(2376): error C2036: 'void *': unknown size
sqlite-vec.c(3530): error C2065: '__FLT_MAX__': undeclared identifier
sqlite-vec.c(3598): error C2065: '__FLT_MAX__': undeclared identifier
sqlite-vec.c(3608): error C2065: '__FLT_MAX__': undeclared identifier
sqlite-vec.c(3840): error C2065: '__FLT_MAX__': undeclared identifier
sqlite-vec.c(5136): error C2375: 'sqlite3_vec_init': redefinition; different linkage
C:\Users\adriankhl\git\test\sqlite\sqlite-vec.h(14): note: see declaration of 'sqlite3_vec_init'
sqlite-vec.c(5149): error C2099: initializer is not a constant
sqlite-vec.c(5151): warning C4047: 'initializing': 'int' differs in levels of indirection from 'char [16]'
sqlite-vec.c(5152): warning C4047: 'initializing': 'int' differs in levels of indirection from 'char [121]'
sqlite-vec.c(5153): warning C4047: 'initializing': 'int' differs in levels of indirection from 'void *'
sqlite-vec.c(5154): warning C4047: 'initializing': 'int' differs in levels of indirection from 'void *'
sqlite-vec.c(5155): warning C4047: 'initializing': 'int' differs in levels of indirection from 'void *'
sqlite-vec.c(5156): warning C4047: 'initializing': 'int' differs in levels of indirection from 'void *'
sqlite-vec.c(5157): warning C4047: 'initializing': 'int' differs in levels of indirection from 'void *'
sqlite-vec.c(5158): warning C4047: 'initializing': 'int' differs in levels of indirection from 'void *'
sqlite-vec.c(5159): warning C4047: 'initializing': 'int' differs in levels of indirection from 'void *'
sqlite-vec.c(5160): warning C4047: 'initializing': 'int' differs in levels of indirection from 'void *'
sqlite-vec.c(5161): warning C4047: 'initializing': 'int' differs in levels of indirection from 'void *'
sqlite-vec.c(5162): warning C4047: 'initializing': 'int' differs in levels of indirection from 'void *'
sqlite-vec.c(5163): warning C4047: 'initializing': 'int' differs in levels of indirection from 'void *'
sqlite-vec.c(5164): warning C4047: 'initializing': 'int' differs in levels of indirection from 'void *'
sqlite-vec.c(5165): warning C4047: 'initializing': 'int' differs in levels of indirection from 'void *'
sqlite-vec.c(5166): warning C4047: 'initializing': 'int' differs in levels of indirection from 'void *'
sqlite-vec.c(5167): warning C4047: 'initializing': 'int' differs in levels of indirection from 'void *'
sqlite-vec.c(5227): error C2375: 'sqlite3_vec_fs_read_init': redefinition; different linkage
C:\Users\adriankhl\git\test\sqlite\sqlite-vec.h(16): note: see declaration of 'sqlite3_vec_fs_read_init'

10c8 commented 4 months ago

I guess you use mingw to compile for windows, because clang-cl works

I can confirm that I had success compiling it on Windows under MinGW installed with MSYS, though the resulting DLL fails to load in the same way.

[...] are you on 64-bit or 32-bit version of windows? Can you also share the Python version + SQLite version your python install has?

Python 3.10.8 on 64-bit Windows 11, SQLite 3.37.2.

asg017 commented 3 months ago

@10c8 @Adriankhl I just made a bunch of fixes to sqlite-vec, so it should support windows + cl.exe, as of v0.0.1-alpha.19.

Could you try this dll and tell me if it works as expected? https://github.com/asg017/sqlite-vec/releases/download/v0.0.1-alpha.19/vec0.dll

Adriankhl commented 2 months ago

When I run

import sqlite3
import sqlite_vec

db = sqlite3.connect(":memory:")
db.enable_load_extension(True)
sqlite_vec.load(db)

I am getting

---------------------------------------------------------------------------
OperationalError                          Traceback (most recent call last)
Cell In[3], line 3
      1 db = sqlite3.connect(":memory:")
      2 db.enable_load_extension(True)
----> 3 sqlite_vec.load(db)
      4 db.enable_load_extension(False)
      5 print(db.execute("select vec_version()").fetchone())

File ~\scoop\apps\python311\current\Lib\site-packages\sqlite_vec\__init__.py:13, in load(conn)
     12 def load(conn: sqlite3.Connection)  -> None:
---> 13   conn.load_extension(loadable_path())

OperationalError: The specified module could not be found.

Similar error happens when I run

db.load_extension("./vec0.dll")

andrevanzuydam commented 1 month ago

Confirmed as working on this particular version, are the current pre-releases supposed to have this fix in ?

@10c8 @Adriankhl I just made a bunch of fixes to sqlite-vec, so it should support windows + cl.exe, as of v0.0.1-alpha.19.

Could you try this dll and tell me if it works as expected? https://github.com/asg017/sqlite-vec/releases/download/v0.0.1-alpha.19/vec0.dll

asg017 commented 1 month ago

@andrevanzuydam does this one work for you? https://github.com/asg017/sqlite-vec/releases/download/v0.1.2-alpha.5/sqlite-vec-0.1.2-alpha.5-loadable-windows-x86_64.tar.gz

Also: what environment are you trying this in (Programming language, OS, client library, etc.)? If you're using Python - how did you install python? We've seen some issues specifically with the Python installation from the Microsoft Store, so curious if you're using that one

andrevanzuydam commented 1 month ago

Thanks for the quick reply Alex, I am using stock standard Python 11 from windows with virtual environment. Everything worked properly on Mac (M3), brew install of python. I'd be happy to assist with the builds if you need help... I just need to get my head around your build process (which looks simple enough). Unfortunatelly that build you linked above is not working. In order to test I replaced sqlite dlls in the distribution until I found this thread and the alpha.19 worked and loaded. The error I get is "Error loading extension, module could not be found". I did test through a bunch of your alpha builds.

asg017 commented 1 month ago

@andrevanzuydam Thanks for the respose!

When you say "stock standard Python 11 from windows", is that from python.org/downloads or from the Microsoft Store?

Could you also run the following script and share the output?

import distutils
import platform
import psutil
import sys

# Distutils info
try:
    distutils_info = distutils.__version__
except AttributeError:
    distutils_info = "distutils version information not available"

# CPU info
cpu_info = {
    "Architecture": platform.architecture(),
    "Machine": platform.machine(),
    "Processor": platform.processor(),
    "Cores (Physical)": psutil.cpu_count(logical=False),
    "Cores (Logical)": psutil.cpu_count(logical=True)
}

# Python installation info
python_info = {
    "Python Version": platform.python_version(),
    "Python Compiler": platform.python_compiler(),
    "Python Build": platform.python_build(),
    "Python Implementation": platform.python_implementation(),
    "Python Executable": sys.executable
}

print("Distutils Info:", distutils_info)
print("CPU Info:", cpu_info)
print("Python Installation Info:", python_info)

andrevanzuydam commented 1 month ago

Yes , from the MS store

[image: http://www.daniweb.com/profiles/1108517/Andre-van-Zuydam] http://www.daniweb.com/profiles/1108517/Andre-van-Zuydam

[image: https://www.openhub.net/accounts/75973?ref=Detailed] https://www.openhub.net/accounts/75973?ref=Detailed

On Mon, 12 Aug 2024 at 09:15, Alex Garcia @.***> wrote:

@andrevanzuydam https://github.com/andrevanzuydam Thanks for the respose!

When you say "stock standard Python 11 from windows", is that from python.org/downloads https://www.python.org/downloads/ or from the Microsoft Store https://apps.microsoft.com/detail/9pjpw5ldxlz5?hl=en-US&gl=US?

Could you also run the following script and share the output?

import distutilsimport platformimport psutilimport sys

Distutils infotry:
distutils_info = distutils.__version__except AttributeError:
distutils_info = "distutils version information not available"
CPU infocpu_info = {
"Architecture": platform.architecture(),
"Machine": platform.machine(),
"Processor": platform.processor(),
"Cores (Physical)": psutil.cpu_count(logical=False),
"Cores (Logical)": psutil.cpu_count(logical=True)
}

Python installation infopython_info = {
"Python Version": platform.python_version(),
"Python Compiler": platform.python_compiler(),
"Python Build": platform.python_build(),
"Python Implementation": platform.python_implementation(),
"Python Executable": sys.executable
} print("Distutils Info:", distutils_info)print("CPU Info:", cpu_info)print("Python Installation Info:", python_info)

— Reply to this email directly, view it on GitHub https://github.com/asg017/sqlite-vec/issues/13#issuecomment-2283256387, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOR7HIORHBJ3DZ7NG73G5LZRBOHRAVCNFSM6AAAAABISURTCSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBTGI2TMMZYG4 . You are receiving this because you were mentioned.Message ID: @.***>

andrevanzuydam commented 1 month ago

Distutils Info: 3.11.8 CPU Info: {'Architecture': ('64bit', 'WindowsPE'), 'Machine': 'AMD64', 'Processor': 'Intel64 Family 6 Model 191 Stepping 2, GenuineIntel', 'Cores (Physical)': 16, 'Cores (Logical)': 24} Python Installation Info: {'Python Version': '3.11.8', 'Python Compiler': 'MSC v.1937 64 bit (AMD64)', 'Python Build': ('tags/v3.11.8:db85d51', 'Feb 6 2024 22:03:32'), 'Python Implementation': 'CPython', 'Python Executable': 'D:\projects\python\simone2\ThoughtDB\.venv\Scripts\python.exe'}

[image: http://www.daniweb.com/profiles/1108517/Andre-van-Zuydam] http://www.daniweb.com/profiles/1108517/Andre-van-Zuydam

[image: https://www.openhub.net/accounts/75973?ref=Detailed] https://www.openhub.net/accounts/75973?ref=Detailed

On Mon, 12 Aug 2024 at 09:15, Alex Garcia @.***> wrote:

@andrevanzuydam https://github.com/andrevanzuydam Thanks for the respose!

When you say "stock standard Python 11 from windows", is that from python.org/downloads https://www.python.org/downloads/ or from the Microsoft Store https://apps.microsoft.com/detail/9pjpw5ldxlz5?hl=en-US&gl=US?

Could you also run the following script and share the output?

import distutilsimport platformimport psutilimport sys

Distutils infotry:
distutils_info = distutils.__version__except AttributeError:
distutils_info = "distutils version information not available"
CPU infocpu_info = {
"Architecture": platform.architecture(),
"Machine": platform.machine(),
"Processor": platform.processor(),
"Cores (Physical)": psutil.cpu_count(logical=False),
"Cores (Logical)": psutil.cpu_count(logical=True)
}

Python installation infopython_info = {
"Python Version": platform.python_version(),
"Python Compiler": platform.python_compiler(),
"Python Build": platform.python_build(),
"Python Implementation": platform.python_implementation(),
"Python Executable": sys.executable
} print("Distutils Info:", distutils_info)print("CPU Info:", cpu_info)print("Python Installation Info:", python_info)

— Reply to this email directly, view it on GitHub https://github.com/asg017/sqlite-vec/issues/13#issuecomment-2283256387, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOR7HIORHBJ3DZ7NG73G5LZRBOHRAVCNFSM6AAAAABISURTCSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBTGI2TMMZYG4 . You are receiving this because you were mentioned.Message ID: @.***>

inktan commented 1 month ago

https://github.com/asg017/sqlite-vec/releases/download/v0.0.1-alpha.19/vec0.dll This v0.0.1-alpha.19 can run normally，thank you However, using pip install sqlite-vec to install the default library may cause issues：OperationalError: The specified module could not be found. I hope the author can update an official library as soon as possible

inktan commented 1 month ago

My computer system is Win11， Distutils Info: 3.12.4 CPU Info: {'Architecture': ('64bit', 'WindowsPE'), 'Machine': 'AMD64', 'Processor': 'Intel64 Family 6 Model 158 Stepping 10, GenuineIntel', 'Cores (Physical)': 6, 'Cores (Logical)': 6}
Python Installation Info: {'Python Version': '3.12.4', 'Python Compiler': 'MSC v.1929 64 bit (AMD64)', 'Python Build': ('main', 'Jun 18 2024 15:03:56'), 'Python Implementation': 'CPython', 'Python Executable': 'c:\Users\wa.n\.conda\envs\pytorch\python.exe'}

andrevanzuydam commented 1 month ago

@inktan yes, I have experienced the same issue, currently I just packaged the alpha.19 in my project and loaded it myself without the library if that will assist you in testing.

inktan commented 1 month ago

@inktan yes, I have experienced the same issue, currently I just packaged the alpha.19 in my project and loaded it myself without the library if that will assist you in testing.

@andrevanzuydam @asg017 We hope to solve this problem as soon as possible. We really like this vector database and thank the author.

jimkring commented 1 month ago

For what it's worth, I was having similar error messages. I "fixed" it by installing mingw using chocolatey (choco install mingw) and rebooted.

tkswanson commented 4 weeks ago

Just wanted to chime in and say I had the same issue with the precompiled .dll, Windows 11 Pro, Python 3.11.7, SQLite 3.41.2 But this version worked for me too https://github.com/asg017/sqlite-vec/releases/download/v0.0.1-alpha.19/vec0.dll

asg017 commented 3 weeks ago

@tkswanson (and anyone else who's having Python/windows issues): can you try v0.1.2-alpha.9 and let me know if it works?

I've completely moved off mingw and now use the msvc compiler starting with this release, so if it works, I'll publish a 0.1.2 with the change.

jimkring commented 3 weeks ago

Thanks for making that change! I’ll give it a test and let you know.

liunux4odoo commented 3 weeks ago

same error with version 0.1.1

Distutils Info: 3.10.13
CPU Info: {'Architecture': ('64bit', 'WindowsPE'), 'Machine': 'AMD64', 'Processor': 'Intel64 Family 6 Model 165 Stepping 3, GenuineIntel', 'Cores (Physical)': 6, 'Cores (Logical)': 12}
Python Installation Info: {'Python Version': '3.10.13', 'Python Compiler': 'MSC v.1916 64 bit (AMD64)', 'Python Build': ('main', 'Sep 11 2023 13:24:38'), 'Python Implementation': 'CPython', 'Python Executable': 'D:\\miniconda3\\envs\\test\\python.exe'}

The error disappeared after replacing vec0.dll by 0.1.2-alpha.9. Expecting version 0.1.2

asg017 / sqlite-vec

Pre-compiled loadable extension won't load on Python (3.10.8, Win11) #13

Distutils infotry:

CPU infocpu_info = {

Python installation infopython_info = {

Distutils infotry:

CPU infocpu_info = {

Python installation infopython_info = {