Crashes on some windows machines when inserting

[l0o0]

Hi, I encountered a crush when I tried to insert a row. I was following the example code. Below is my python code.

import vectorlite_py
from sqlalchemy import create_engine, text
engine = create_engine("sqlite:///test.db")
conn = engine.raw_connection()
conn.enable_load_extension(True)
conn.load_extension(vectorlite_py.vectorlite_path())
r = conn.execute("select vectorlite_info(); ")
r.fetchall()
# [('vectorlite extension version 0.2.0, built with SSE',)]
sql = """select vector_distance(vector_from_json('[1,2,3]'), vector_from_json('[3,4,5]'), 'l2');"""
r = conn.execute(sql)
r.fetchall()
# Out[21]: [(12.0,)]
sql = """create virtual table my_table using vectorlite(my_embedding float32[3], hnsw(max_elements=100));"""
r = conn.execute(sql)
sql = """insert into my_table(rowid, my_embedding) values (0, vector_from_json('[1,2,3]'));"""
r = conn.execute(sql)  # crushed after this , my_table was empty

Meanwhile, I downloaded Sqlite3 and tested the demo code in PowerShell. It crushed after this insert statement.

• System: Win10
• CPU: i5-8250U
• Memory: 16G

Is there anything else I can provide to help you resolve this issue?

[1yefuwang1]

Thank you for reporting this issue. But I can't repro it on Windows or WSL

[1yefuwang1]

Do you mind providing more info on how it crashes? Is it a segfault or an unhandled python exception?

Besides, which version are you using?

[l0o0]

@1yefuwang1 Thank you for your reply.

Is it a segfault or an unhandled python exception? As the screenshot, when I run r = conn.execute(sql), the IPython console existed without any error messages.

which version are you using?

Vectorlite version: vectorlite extension version 0.2.0, built with SSE. Python 3.12.4 vectorlite_py 0.2.0

I don't know much about SQLite logging. Perhaps you could guide me in gathering more information about this crush.

[1yefuwang1]

To help me troubleshoot, do you mind to

1. print something after the execution of the insert statement and see whether it gets printed.
2. run your script using python directly instead of IPython and see what happens

[l0o0]

@1yefuwang1 I updated the test.py code. The screenshot is the output. Nothing is printed after the insert.

[1yefuwang1]

It's kind of weird that your script just silently exits without reporting any errors.

@l0o0 Are you sure that your print statement was saved before running?

What happens if you execute your sql commands in a sqlite3 CLI shell?

[1yefuwang1]

@l0o0 Which sqlite/sqlalchemy version are you using? Do you mind trying again with python's built-in sqlite3 pacakge?

[fukuro031]

I encountered the same issue.

I'm running the vectorlite\examples\quickstart.py sample, and I'm also getting an exception during the insert operation.

Here are the logs I added before and after the insert:

import faulthandler

print("insert starting")

faulthandler.enable()

cursor.executemany('insert into my_table(rowid, my_embedding) values (?, ?)', [(i, data[i].tobytes()) for i in range(NUM_ELEMENTS)])

print("insert done")

Execution output:

System Details: OS: Windows 10 Python: 3.12.7 Vectorlite version: vectorlite extension version 0.2.0, built with SSE APSW version: 3.46.1.0

[1yefuwang1]

Thank you @fukuro031 for reporting. It can be confirmed that it is a memory violation. And both of you are using python3.12. This is indeed very weird because all unit tests,integration tests and examples run successfully on windows.

Do you mind sharing what CPU you are using and whether you can reproduce it on other systems like WSL/Linux if it is convenient for you.

I can't reproduce it with my PC (Win11, python3.10.11, apsw 3.46.1.0) I'll try to reproduce it on Win10 and python3.12.7.

[fukuro031]

@1yefuwang1 My current CPU is an Intel(R) Core(TM) i7-10850H. I also tested on another WIN10 system with a 12th Gen Intel(R) Core(TM) i7-1270P, where the application runs successfully. Could it be related to the SSE instruction set?

[1yefuwang1]

@fukuro031 Thank you for the info. The "built with SSE" message is out of date now. After version 0.2.0, vectorlite uses highway to detect the best available SIMD instruction to use at runtime.