rogerbinns / apsw

Another Python SQLite wrapper
https://rogerbinns.github.io/apsw/
Other
733 stars 97 forks source link

Maintain a module level weakref list of connections #416

Closed rogerbinns closed 1 year ago

rogerbinns commented 1 year ago

If a function is registered then CPython doesn't garbage collect the Connection. Running this code under valgrind --track-fds=yes will show the file handle still open on exit. This happens with both apsw and sqlite3.

if True:
    import apsw
    connection = apsw.Connection("dbfile")
    connection.createscalarfunction("foo", lambda : 0)
else:
    import sqlite3
    connection = sqlite3.connect("dbfilesqlite3")
    connection.create_function("foo", -1, lambda : 0)

If another line is added creating the function again, but with None as the callback, then both apsw and sqlite3 exit without the open connection, showing this is caused by the data structure holding the callbacks meaning CPython doesn't do the gc because it can't find every reference. A quick experiment adding GC support to that data structure didn't work.

It is likely other corner cases may exist too. This proposes that the module keep a weakref list of connections, so they can all be found, introspected, and closed as needed.