search

husio / python-sqlite3-backup

Sqlite3 online API CPython implementation module
http://pypi.python.org/pypi/sqlitebck/
42 stars 9 forks source link

Cannot Be Even Imported If There Are More Than One Versions of SQLite Set Up #10

Open Xtigyro opened 6 years ago

Xtigyro commented 6 years ago 
Traceback (most recent call last):
  File "/opt/rh/python27/root/usr/lib64/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/opt/rh/python27/root/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/opt/IBM/InformationServer/Server/DSEngine/bin/ISMD-emeaprd.py/__main__.py", line 38, in <module>
AttributeError: 'module' object has no attribute 'Connection'
/var/tmp/sclPqKpVb: line 8:  2032 Segmentation fault      (core dumped) python /opt/IBM/InformationServer/Server/DSEngine/bin/ISMD-emeaprd.py

And line 38 is: import sqlitebck

It does NOT even come to the point in my Python script where it actually tries to make an online backup of the DB - which is:

def db_bup():
    """ Backup the ISMon DB and clean too old backups """
    # One DB bup per day at this hour
    bup_time = now.strftime('%HH')
    if bup_time == '10H':
        conn_to_backup = sqlite3.connect(sqldb_loc)
        conn_backup = sqlite3.connect(sqldb_bup_loc)
        sqlitebck.copy(conn_to_backup, conn_backup)
        conn_to_backup.close()
        conn_backup.close()
        print ("DB Backup - finished.")

I've compiled it on a RHEL 6.9 - no errors:

[05.11.2017] [11:13:05 GMT] root@ $ scl enable python27 bash
[root@prod-eur-isbi01 sqlitebck-1.3]# python setup.py build
running build
running build_ext
building 'sqlitebck' extension
creating build
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/src
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/local/include -I/opt/rh/python27/root/usr/include/python2.7 -c src/sqlitebck.c -o build/temp.linux-x86_64-2.7/src/sqlitebck.o
In file included from /opt/rh/python27/root/usr/include/python2.7/pyconfig.h:6,
                 from /opt/rh/python27/root/usr/include/python2.7/Python.h:8,
                 from src/sqlitebck.c:33:
/opt/rh/python27/root/usr/include/python2.7/pyconfig-64.h:1199:1: warning: "_POSIX_C_SOURCE" redefined
In file included from /usr/include/string.h:27,
                 from src/sqlitebck.c:31:
/usr/include/features.h:162:1: warning: this is the location of the previous definition
In file included from /opt/rh/python27/root/usr/include/python2.7/pyconfig.h:6,
                 from /opt/rh/python27/root/usr/include/python2.7/Python.h:8,
                 from src/sqlitebck.c:33:
/opt/rh/python27/root/usr/include/python2.7/pyconfig-64.h:1221:1: warning: "_XOPEN_SOURCE" redefined
In file included from /usr/include/string.h:27,
                 from src/sqlitebck.c:31:
/usr/include/features.h:164:1: warning: this is the location of the previous definition
creating build/lib.linux-x86_64-2.7
gcc -pthread -shared build/temp.linux-x86_64-2.7/src/sqlitebck.o -L/opt/rh/python27/root/usr/lib64 -lsqlite3 -lpython2.7 -o build/lib.linux-x86_64-2.7/sqlitebck.so
[root@ lib.linux-x86_64-2.7]# ll
total 28
-rwxr-xr-x 1 root root 24789 Nov  5 11:13 sqlitebck.so

The install process also is fine:

python setup.py install
running install
running build
running build_ext
running install_lib
copying build/lib.linux-x86_64-2.7/sqlitebck.so -> /opt/rh/python27/root/usr/lib64/python2.7/site-packages
running install_egg_info
Writing /opt/rh/python27/root/usr/lib64/python2.7/site-packages/sqlitebck-1.3-py2.7.egg-info

And I've used for several months ver. 1.2.1 - but now after upgrading to SQLite 3.21 - it says that it cannot find an SQLite database:

Traceback (most recent call last):
  File "/opt/rh/python27/root/usr/lib64/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/opt/rh/python27/root/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/opt/IBM/InformationServer/Server/DSEngine/bin/ISMD-emeaprd.py/__main__.py", line 2352, in <module>
  File "/opt/IBM/InformationServer/Server/DSEngine/bin/ISMD-emeaprd.py/__main__.py", line 554, in db_bup
TypeError: Given source object is not a sqlite database

EDIT_1: Tried in interactive mode:

bash-4.1$ python
Python 2.7.13 (default, Feb  8 2017, 06:30:30)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-16)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>>
>>> import sqlitebck
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'Connection'

Using the latest 'pip' leads to the same result:

[root@localhost sqlitebck-1.3]# pip install sqlitebck
Collecting sqlitebck
Installing collected packages: sqlitebck
Successfully installed sqlitebck-1.3
[root@localhost sqlitebck-1.3]#
[root@localhost sqlitebck-1.3]#
[root@localhost sqlitebck-1.3]#
[root@localhost sqlitebck-1.3]# python
Python 2.7.13 (default, Apr 12 2017, 06:53:51)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-18)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlitebck
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'Connection'

EDIT_2: So it appears the problem is related to the other older instances of SQLite - I've created a new virtual machine with CentOS 6.9 and compiled and installed manually only SQLite 3.21. Then compiled 'sqlitebck' ver. 1.3 - it now works - but I'm wondering how I can fix this on the other machines I have where I cannot start from scratch.

What files/dependencies does 'sqlitebck' expect? Because even after compiling and installing SQLite 3.21 using the defaults on the other (old) machines - the error is still the same (even though I've added most of the paths of the new SQLite 3.21 in the LD_LIBRARY_PATH).

And is there a way to include statically the files of SQLite 3.21 so only 'sqlitebck' can see them as the default ones on the system?

For example, I've compiled 'pysqlite' ver. 2.8.3 with built-in files of SQLite 3.21 using: python setup.py build_static

Is there a way to achieve the same thing with 'sqlitebck'?

Xtigyro commented 6 years ago

1) Downloaded the latest SQLite - ver. 3.21 (sqlite-autoconf-3210000.tar.gz). 2) Compiled and installed it.

[root@localhost sqlitebck]# sqlite3 -version 3.21.0 2017-10-24 18:55:49 1a584e499906b5c87ec7d43d4abce641fdf017c42125b083109bc77c4de48827

3) Make sure to link its libraries in your session and also to make available its INCLUDE files ('.h' files) - then recompiled 'sqlitebck' ver. 1.3.

either use libtool, and specify the full pathname of the library, or use the '-LLIBDIR' flag during linking and do at least one of the following:

It works now.

[root@localhost sqlitebck-1.3]# python Python 2.7.13 (default, Apr 12 2017, 06:56:55) [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux2 Type "help", "copyright", "credits" or "license" for more information.

import sqlitebck

Xtigyro commented 6 years ago

I believe we need to find a way of using version 1.3 with an older SQLite installed - that is to say, without the need of upgrading the SQLite on the machine.

Xtigyro commented 6 years ago

Also - the above solution works only on RHEL/CentOS ver. 7 or above.

Xtigyro commented 6 years ago

So it appears the problem is related to the other older instances of SQLite - I've created a new virtual machine with CentOS 6.9 and compiled and installed manually only SQLite 3.21. Then compiled 'sqlitebck' ver. 1.3 - it now works - but I'm wondering how I can fix this on the other machines I have where I cannot start from scratch.

What files/dependencies does 'sqlitebck' expect? Because even after compiling and installing SQLite 3.21 using the defaults on the other (old) machines - the error is still the same (even though I've added most of the paths of the new SQLite 3.21 in the LD_LIBRARY_PATH).

Xtigyro commented 6 years ago

And is there a way to include statically the files of SQLite 3.21 so only 'sqlitebck' can see them as the default ones on the system?

For example, I've compiled 'pysqlite' ver. 2.8.3 with built-in files of SQLite 3.21 using: python setup.py build_static

Is there a way to achieve the same thing with 'sqlitebck'?

husio commented 6 years ago

sqlitebck expects ABI compability with python's sqlite module, because of included for simplicity files. It probably requires some decent version of SQLlite as well and that is expected to be provided by the system.

If the problem is that system provides old version of SQLite and you would like to use your own build, right now I can only suggest updating setup.py file with extra instructions. From what I understand, you would like to use non standard (not provided by the system) version of SQLite.

I can try to help you with the changes and make sure they work for systems with updated version of sqlite, but I am not sure how can I assist you with solving your particular problem.

Xtigyro commented 6 years ago

Hi Piotr - yes, it appears exactly this - after updating 'pysqlite' to ver. 2.8.3 (compiled with a static built-in of SQLite 3.21)... 'sqlitebck' stopped working. Trying to update it leads to the narrative/adventures above.

Generally - I use RHEL/CentOS 6.9 or 7.3 (most of the machines run on 6.9 'Santiago'). In order to upgrade SQLite to 3.21 in Python 2.7 (that comes with RedHat Software Compilations) - I did this - https://hadzhiev.com/2017/11/01/upgrade-sqlite-in-python/ (my blog).

I think that if we can do some changes to 'setup.py' - will be very nice. I would love to learn more complex Python stuff. You can find me on Skype (miroslav.hadzhiev) if you would like to discuss this in details using a chat.

Thank you and good night!

Miro