search

ComparativeGenomicsToolkit / cactus

Official home of genome aligner based upon notion of Cactus graphs
Other
529 stars 111 forks source link

No module named '_sqlite3' #773

Open rsharris opened 2 years ago

rsharris commented 2 years ago

( Note: I'm trying the NON-legacy binary on different machine than I was using for issue #762 )

I'm attempting to run the v2.2.0 pre-compiled binaries (cactus-bin-v2.2.0.tar.gz). I'm running a very small test case with five 50Kbp "genomes". This test case had run successfully on a different machine using the legacy-v2.1.1 version of cactus.

Cactus fails almost immediately, with the python exception ModuleNotFoundError: No module named '_sqlite3' . I've appended the full log at the end of this post.

My guess, after doing some googling, is this is a problem with how my python was built. I had to build my own version of python since the system python on this machine is only 3.6. I built 3.8.13, without using admin privs nor sudo.

Googling for that error message, I found this discussion: https://stackoverflow.com/questions/1210664/no-module-named-sqlite3 . That thread started 13 years ago so it might not be relevant. And many of the suggestions in that thread seem as useful as drinking anti-freeze. But about 25% down that thread there is a suggestion that I might have needed to do ./configure --enable-loadable-sqlite-extensions --enable-optimizations when I began building my version of python.

Any thoughts on whether that's the right solution? I'm reticent to "just try it" since I could end up chasing the wild goose.

I also tried searching for sqlite in cactus issues and in toil issues (assuming DataBioshere/toil is the right toil), but nothing seemed relevant.

Bob H

e5-cse-cbmedg01:fiddling.20220822B$ source ${cactus_env}/bin/activate (cactus.20220822.env) e5-cse-cbmedg01:fiddling.20220822B$ time cactus jobstore seqFile.20220822B.txt seqFile.20220822B.hal Traceback (most recent call last): File "/research/rsh915/development/third_party/cactus/cactus-bin-v2.2.0/cactus.20220822.env/bin/cactus", line 33, in sys.exit(load_entry_point('Cactus==2.2.0', 'console_scripts', 'cactus')()) File "/research/rsh915/development/third_party/cactus/cactus-bin-v2.2.0/cactus.20220822.env/bin/cactus", line 25, in importlib_load_entry_point return next(matches).load() File "/research/rsh915/development/third_party/python3/installs/linux-x86_64/lib/python3.8/importlib/metadata.py", line 77, in load module = import_module(match.group('module')) File "/research/rsh915/development/third_party/python3/installs/linux-x86_64/lib/python3.8/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1014, in _gcd_import File "", line 991, in _find_and_load File "", line 975, in _find_and_load_unlocked File "", line 671, in _load_unlocked File "", line 843, in exec_module File "", line 219, in _call_with_frames_removed File "/research/rsh915/development/third_party/cactus/cactus-bin-v2.2.0/cactus.20220822.env/lib/python3.8/site-packages/cactus/progressive/cactus_progressive.py", line 19, in from toil.lib.bioio import getTempFile File "/research/rsh915/development/third_party/cactus/cactus-bin-v2.2.0/cactus.20220822.env/lib/python3.8/site-packages/toil/init.py", line 30, in from toil.lib.retry import retry File "/research/rsh915/development/third_party/cactus/cactus-bin-v2.2.0/cactus.20220822.env/lib/python3.8/site-packages/toil/lib/retry.py", line 130, in import sqlite3 File "/research/rsh915/development/third_party/python3/installs/linux-x86_64/lib/python3.8/sqlite3/init.py", line 23, in from sqlite3.dbapi2 import File "/research/rsh915/development/third_party/python3/installs/linux-x86_64/lib/python3.8/sqlite3/dbapi2.py", line 27, in from _sqlite3 import ModuleNotFoundError: No module named '_sqlite3'

diekhans commented 2 years ago

Hi Bob,

Yes, this is python not having sqlite3 support compiled in.

I end up compiling much of what I use without room, since I am on a time share with ancient software.

So I compile both sqlite3 and the latest python. Here is what I use, with MED_OPT being set to the prefix directory for all my software (${HOME}/opt/):

(LDFLAGS="-L${MED_OPT}/lib -Wl,-rpath -Wl,${MED_OPT}/lib" CPPFLAGS=-I${MED_OPT}/include PKG_CONFIG_PATH="${MED_OPT}/lib/pkgconfig:${MED_OPT}/share/pkgconfig:/usr/share/pkgconfig:/usr/lib/pkgconfig" ./configure --prefix=${MED_OPT} && nice make -j $(nproc) && make install)>&build.out &

Feel free to mail me directly if you need more help

Mark

Bob Harris @.***> writes:

( Note: I'm trying the NON-legacy binary on different machine than I was using for issue #762 )

I'm attempting to run the v2.2.0 pre-compiled binaries (cactus-bin-v2.2.0.tar.gz). I'm running a very small test case with five 50Kbp "genomes". This test case had run successfully on a different machine using the legacy-v2.1.1 version of cactus.

Cactus fails almost immediately, with the python exception ModuleNotFoundError: No module named '_sqlite3' . I've appended the full log at the end of this post.

My guess, after doing some googling, is this is a problem with how my python was built. I had to build my own version of python since the system python on this machine is only 3.6. I built 3.8.13, without using admin privs nor sudo.

Googling for that error message, I found this discussion: https://stackoverflow.com/questions/1210664/no-module-named-sqlite3 . That thread started 13 years ago so it might not be relevant. And many of the suggestions in that thread seem as useful as drinking anti-freeze. But about 25% down that thread there is a suggestion that I might have needed to do ./configure --enable-loadable-sqlite-extensions --enable-optimizations when I began building my version of python.

Any thoughts on whether that's the right solution? I'm reticent to "just try it" since I could end up chasing the wild goose.

I also tried searching for sqlite in cactus issues and in toil issues (assuming DataBioshere/toil is the right toil), but nothing seemed relevant.

Bob H

e5-cse-cbmedg01:fiddling.20220822B$ source ${cactus_env}/bin/activate (cactus.20220822.env) e5-cse-cbmedg01:fiddling.20220822B$ time cactus jobstore seqFile.20220822B.txt seqFile.20220822B.hal Traceback (most recent call last): File "/research/rsh915/development/third_party/cactus/cactus-bin-v2.2.0/cactus.20220822.env/bin/cactus", line 33, in sys.exit(load_entry_point('Cactus==2.2.0', 'console_scripts', 'cactus')()) File "/research/rsh915/development/third_party/cactus/cactus-bin-v2.2.0/cactus.20220822.env/bin/cactus", line 25, in importlib_load_entry_point return next(matches).load() File "/research/rsh915/development/third_party/python3/installs/linux-x86_64/lib/python3.8/importlib/metadata.py", line 77, in load module = import_module(match.group('module')) File "/research/rsh915/development/third_party/python3/installs/linux-x86_64/lib/python3.8/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1014, in _gcd_import File "", line 991, in _find_and_load File "", line 975, in _find_and_load_unlocked File "", line 671, in _load_unlocked File "", line 843, in exec_module File "", line 219, in _call_with_frames_removed File "/research/rsh915/development/third_party/cactus/cactus-bin-v2.2.0/cactus.20220822.env/lib/python3.8/site-packages/cactus/progressive/cactus_progressive.py", line 19, in from toil.lib.bioio import getTempFile File "/research/rsh915/development/third_party/cactus/cactus-bin-v2.2.0/cactus.20220822.env/lib/python3.8/site-packages/toil/init.py", line 30, in from toil.lib.retry import retry File "/research/rsh915/development/third_party/cactus/cactus-bin-v2.2.0/cactus.20220822.env/lib/python3.8/site-packages/toil/lib/retry.py", line 130, in import sqlite3 File "/research/rsh915/development/third_party/python3/installs/linux-x86_64/lib/python3.8/sqlite3/init.py", line 23, in from sqlite3.dbapi2 import File "/research/rsh915/development/third_party/python3/installs/linux-x86_64/lib/python3.8/sqlite3/dbapi2.py", line 27, in from _sqlite3 import ModuleNotFoundError: No module named '_sqlite3'

-- Reply to this email directly or view it on GitHub: https://github.com/ComparativeGenomicsToolkit/cactus/issues/773 You are receiving this because you are subscribed to this thread.

Message ID: @.***>

rsharris commented 2 years ago

Here's a recap, in case anyone else runs into the same problem.

The problem was that my python didn't have compiled-in support for some module cactus needs. These include sqlite3 and bz2 for certain. And perhaps ctypes and uuid.

The python build process will automatically compile these in, but only if the necessary libraries exist on your machine at the time python is built. So I need to install several libraries and rebuild python.