search

rogerbinns / apsw

Another Python SQLite wrapper
https://rogerbinns.github.io/apsw/
Other
733 stars 97 forks source link

Tests fail with AttributeError: module 'apsw' has no attribute 'format_sql_value' #462

Closed mcepl closed 1 year ago

mcepl commented 1 year ago

When running the test suite as part of packaging your module for openSUSE, we have encountered this error:

[   17s] + python3.9 -m unittest -v apsw/tests.py
[   17s] Traceback (most recent call last):
[   17s]   File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
[   17s]     return _run_code(code, main_globals, None,
[   17s]   File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
[   17s]     exec(code, run_globals)
[   17s]   File "/usr/lib64/python3.9/unittest/__main__.py", line 18, in <module>
[   17s]     main(module=None)
[   17s]   File "/usr/lib64/python3.9/unittest/main.py", line 100, in __init__
[   17s]     self.parseArgs(argv)
[   17s]   File "/usr/lib64/python3.9/unittest/main.py", line 147, in parseArgs
[   17s]     self.createTests()
[   17s]   File "/usr/lib64/python3.9/unittest/main.py", line 158, in createTests
[   17s]     self.test = self.testLoader.loadTestsFromNames(self.testNames,
[   17s]   File "/usr/lib64/python3.9/unittest/loader.py", line 220, in loadTestsFromNames
[   17s]     suites = [self.loadTestsFromName(name, module) for name in names]
[   17s]   File "/usr/lib64/python3.9/unittest/loader.py", line 220, in <listcomp>
[   17s]     suites = [self.loadTestsFromName(name, module) for name in names]
[   17s]   File "/usr/lib64/python3.9/unittest/loader.py", line 154, in loadTestsFromName
[   17s]     module = __import__(module_name)
[   17s]   File "/home/abuild/rpmbuild/BUILD/apsw-3.42.0.1/apsw/tests.py", line 33, in <module>
[   17s]     import apsw.shell
[   17s]   File "/home/abuild/rpmbuild/BUILD/apsw-3.42.0.1/apsw/shell.py", line 29, in <module>
[   17s]     class Shell:
[   17s]   File "/home/abuild/rpmbuild/BUILD/apsw-3.42.0.1/apsw/shell.py", line 2877, in Shell
[   17s]     def write_value(self, value, fmt=apsw.format_sql_value):
[   17s] AttributeError: module 'apsw' has no attribute 'format_sql_value'

Complete build log with all details of packages used and steps taken to reproduce.

rogerbinns commented 1 year ago

I'm looking into this, but here are some notes while I do that.

mcepl commented 1 year ago

I'm looking into this, but here are some notes while I do that.

Oh, it was a stupid mistake. I was setting PYTHONPATH to non-arch directories, and apsw is arch-package. With correct macro used I am down to one failure and three errors:

[   53s] ======================================================================
[   53s] ERROR: testLoadExtension (apsw.tests.APSW)
[   53s] Check loading of extensions
[   53s] ----------------------------------------------------------------------
[   53s] Traceback (most recent call last):
[   53s]   File "/home/abuild/rpmbuild/BUILDROOT/python-apsw-3.42.0.1-0.x86_64/usr/lib64/python3.9/site-packages/apsw/tests.py", line 3406, in testLoadExtension
[   53s]     self.db.loadextension(LOADEXTENSIONFILENAME)
[   53s] apsw.ExtensionLoadingError: ExtensionLoadingError: ./testextension.sqlext.so: cannot open shared object file: No such file or directory
[   53s] 
[   53s] ======================================================================
[   53s] ERROR: testVFS (apsw.tests.APSW)
[   53s] Verify VFS functionality
[   53s] ----------------------------------------------------------------------
[   53s] Traceback (most recent call last):
[   53s]   File "/home/abuild/rpmbuild/BUILDROOT/python-apsw-3.42.0.1-0.x86_64/usr/lib64/python3.9/site-packages/apsw/tests.py", line 6762, in testVFS
[   53s]     testdb()
[   53s]   File "/home/abuild/rpmbuild/BUILDROOT/python-apsw-3.42.0.1-0.x86_64/usr/lib64/python3.9/site-packages/apsw/tests.py", line 10254, in vfstestdb
[   53s]     db.loadextension(LOADEXTENSIONFILENAME)
[   53s] apsw.ExtensionLoadingError: ExtensionLoadingError: ./testextension.sqlext.so: cannot open shared object file: No such file or directory
[   53s] 
[   53s] ======================================================================
[   53s] ERROR: testVFSWithWAL (apsw.tests.APSW)
[   53s] Verify VFS using WAL
[   53s] ----------------------------------------------------------------------
[   53s] Traceback (most recent call last):
[   53s]   File "/home/abuild/rpmbuild/BUILDROOT/python-apsw-3.42.0.1-0.x86_64/usr/lib64/python3.9/site-packages/apsw/tests.py", line 6009, in testVFSWithWAL
[   53s]     self.testVFS()
[   53s]   File "/home/abuild/rpmbuild/BUILDROOT/python-apsw-3.42.0.1-0.x86_64/usr/lib64/python3.9/site-packages/apsw/tests.py", line 6762, in testVFS
[   53s]     testdb()
[   53s]   File "/home/abuild/rpmbuild/BUILDROOT/python-apsw-3.42.0.1-0.x86_64/usr/lib64/python3.9/site-packages/apsw/tests.py", line 10254, in vfstestdb
[   53s]     db.loadextension(LOADEXTENSIONFILENAME)
[   53s] apsw.ExtensionLoadingError: ExtensionLoadingError: ./testextension.sqlext.so: cannot open shared object file: No such file or directory
[   53s] 
[   53s] ======================================================================
[   53s] FAIL: testShell (apsw.tests.APSW)
[   53s] Check Shell functionality
[   53s] ----------------------------------------------------------------------
[   53s] Traceback (most recent call last):
[   53s]   File "/home/abuild/rpmbuild/BUILDROOT/python-apsw-3.42.0.1-0.x86_64/usr/lib64/python3.9/site-packages/apsw/tests.py", line 9127, in testShell
[   53s]     isempty(fh[2])
[   53s]   File "/home/abuild/rpmbuild/BUILDROOT/python-apsw-3.42.0.1-0.x86_64/usr/lib64/python3.9/site-packages/apsw/tests.py", line 7669, in isempty
[   53s]     self.assertEqual(get(x), "")
[   53s] AssertionError: 'At Line 14184 : test-shell-in\nExtensionL[1834 chars]re\n' != ''
[   53s] - At Line 14184 : test-shell-in
[   53s] - ExtensionLoadingError: ./testextension.sqlext.so: cannot open shared object file: No such file or directory
[   53s] - 
[   53s] - Frame cmdloop in /home/abuild/rpmbuild/BUILDROOT/python-apsw-3.42.0.1-0.x86_64/usr/lib64/python3.9/site-packages/apsw/shell.py at line 794
[   53s] -    command = '.load ./testextension.sqlext alternate_sqlite3_extension_init'
[   53s] -      intro = 'SQLite version 3.42.0 (APSW 3.42.0.1)\nEnter ".help" for instructions\n'
[   53s] -       self = <apsw.shell.Shell object at 0x7fb8e027aa60>
[   53s] -  transient = None
[   53s] - using_readline = False
[   53s] - 
[   53s] - Frame process_complete_line in /home/abuild/rpmbuild/BUILDROOT/python-apsw-3.42.0.1-0.x86_64/usr/lib64/python3.9/site-packages/apsw/shell.py at line 2963
[   53s] -    command = '.load ./testextension.sqlext alternate_sqlite3_extension_init'
[   53s] -       self = <apsw.shell.Shell object at 0x7fb8e027aa60>
[   53s] - 
[   53s] - Frame process_command in /home/abuild/rpmbuild/BUILDROOT/python-apsw-3.42.0.1-0.x86_64/usr/lib64/python3.9/site-packages/apsw/shell.py at line 1023
[   53s] -        cmd = ['load', './testextension.sqlext', 'alternate_sqlite3_extension_init']
[   53s] -    command = '.load ./testextension.sqlext alternate_sqlite3_extension_init'
[   53s] -         fn = <bound method Shell.command_load of <apsw.shell.Shell object at 0x7fb8e027aa60>>
[   53s] -       self = <apsw.shell.Shell object at 0x7fb8e027aa60>
[   53s] - 
[   53s] - Frame command_load in /home/abuild/rpmbuild/BUILDROOT/python-apsw-3.42.0.1-0.x86_64/usr/lib64/python3.9/site-packages/apsw/shell.py at line 2128
[   53s] -        cmd = ['./testextension.sqlext', 'alternate_sqlite3_extension_init']
[   53s] -       self = <apsw.shell.Shell object at 0x7fb8e027aa60>
[   53s] - 
[   53s] - <class 'apsw.ExtensionLoadingError'>: ExtensionLoadingError('ExtensionLoadingError: ./testextension.sqlext.so: cannot open shared object file: No such file or directory')
[   53s] - SQLError: no such function: doubleup
[   53s] -    select doubleup(2);
[   53s] -           ^--- error here

Updated build log

Apparently testextension.sqlext.so has never been built:

abuild@tumbleweed-pkg:~> find . -name testextension\*
./rpmbuild/BUILD/apsw-3.42.0.1/src/testextension.c
abuild@tumbleweed-pkg:~>

(~/rpmbuild/BUILD/apsw-3.42.0.1 is where the tarball is expanded)

rogerbinns commented 1 year ago

I recommend you update your build process to add a setup.apsw file with these contents:

[build_ext]
use_system_sqlite_config = true

This will ensure that APSW's compilation options match that of the system SQLite it is being compiled and linked against. For example extension loading or extra column metadata cause various SQLite C apis to be present or omitted, which APSW needs to match.

You can tell it has taken effect by the second line appearing early in the build:

SQLite: Using system sqlite include/libraries
Extracting configuration from libsqlite3.so.0
mcepl commented 1 year ago

OK, setup.apsw is in place, it changes the output:

tumbleweed-pkg~/b/d/python-apsw (1M)$ osc lbl|grep 'SQLite: Using'
[   10s]   SQLite: Using system sqlite include/libraries
[   10s]   SQLite: Using include/libraries in sqlite3 subdirectory
[   10s]   SQLite: Using generated /home/abuild/rpmbuild/BUILD/apsw-3.42.0.1/sqlite3/sqlite3config.h
[   13s]   SQLite: Using include/libraries in sqlite3 subdirectory
[   13s]   SQLite: Using generated /home/abuild/rpmbuild/BUILD/apsw-3.42.0.1/sqlite3/sqlite3config.h
[   13s]   SQLite: Using include/libraries in sqlite3 subdirectory
[   13s]   SQLite: Using generated /home/abuild/rpmbuild/BUILD/apsw-3.42.0.1/sqlite3/sqlite3config.h
[   16s]   SQLite: Using include/libraries in sqlite3 subdirectory
[   16s]   SQLite: Using generated /home/abuild/rpmbuild/BUILD/apsw-3.42.0.1/sqlite3/sqlite3config.h
[   16s]   SQLite: Using include/libraries in sqlite3 subdirectory
[   16s]   SQLite: Using generated /home/abuild/rpmbuild/BUILD/apsw-3.42.0.1/sqlite3/sqlite3config.h
tumbleweed-pkg~/b/d/python-apsw (1M)$

(see update complete log)

Not sure, why only one Using system sqlite include/libraries remains.

Anyway, it didn’t change anything about that testextension.sqlext.so.

mcepl commented 1 year ago

Sorry, for not providing updated log immediately.

rogerbinns commented 1 year ago

The extension loading errors are because the test extension is not present. You have three choices:

1: Do testing using python3 -m apsw.tests -v NOT by invoking unittest directly as is currently done 2: Build the test extension too 3: I can hack tests so that if it is invoked by -m unittest it runs the internal routine to delete non-applicable tests.

I recommend doing the first one. It will remove the tests depending on the test extension.

The test extension is trivial. Here are SQLite's instructions on how to compile. There is a also a hidden flag to APSW's setup.py which uses the same compiler and flags as was used to build Python.

python3 setup.py build_test_extension
rogerbinns commented 1 year ago

It is 99% resolved so not an issue not having the log :)

The extension loading and particularly testing code has been unchanged for over a decade and has never had a bug, so it is fine skipping that. I do test it as part of my testing. But for example I don't test it when doing pypi builds.

rogerbinns commented 1 year ago

Not sure, why only one Using system sqlite include/libraries remains

I'll fix that, but it is a cosmetic issue not a behavioural change. It is a side effect of repeatedly building from the same source directory.

mcepl commented 1 year ago

Using $python -m apsw.test -v makes the test suite pass, but I still have to build the test extension separately, otherwise I get this warning:

[  404s] Not doing LoadExtension test.  You need to compile the extension first
[  404s] 
[  404s]   python3 setup.py build_test_extension
[  404s] ok

When I added this to the build section, everything seems to work fine.

rogerbinns commented 1 year ago

Thank you for packaging APSW for openSUSE, and pointing to these issues.