search

jnwatson / py-lmdb

Universal Python binding for the LMDB 'Lightning' Database
http://lmdb.readthedocs.io/
Other
646 stars 106 forks source link

LMDB v < 1.4.1 fails to install with pip #365

Open chkp-shacharah opened 4 months ago

chkp-shacharah commented 4 months ago

Affected Operating Systems

Affected py-lmdb Version

any version > 1.4.1

py-lmdb Installation Method

pip install lmdb

Using bundled or distribution-provided LMDB library?

not sure ..

Machine "free -m" output

$ free -m
              total        used        free      shared  buff/cache   available
Mem:           7750        1516         327          13        5905        5914
Swap:             0           0           0

Other important machine info

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
$ uname -vispor
Linux 5.4.0-1030-aws #31-Ubuntu SMP Fri Nov 13 11:40:37 UTC 2020 x86_64 x86_64 GNU/Linux

Describe Your Problem

Installing the lmdb lib via pip (python2.7) , does not work with versions > 1.4.1

Errors/exceptions Encountered

$ pip install lmdb
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Collecting lmdb
  Using cached lmdb-1.5.1.tar.gz (881 kB)
Using legacy 'setup.py install' for lmdb, since package 'wheel' is not installed.
Installing collected packages: lmdb
    Running setup.py install for lmdb ... error
    ERROR: Command errored out with exit status 1:
     command: /home/shachar/.pyenv/versions/2.7.18/bin/python2.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-kGq0cd/lmdb/setup.py'"'"'; __file__='"'"'/tmp/pip-install-kGq0cd/lmdb/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-zQBrCU/install-record.txt --single-version-externally-managed --compile --install-headers /home/shachar/.pyenv/versions/2.7.18/include/python2.7/lmdb
         cwd: /tmp/pip-install-kGq0cd/lmdb/
    Complete output (31 lines):
    py-lmdb: Using bundled liblmdb with py-lmdb patches; override with LMDB_FORCE_SYSTEM=1 or LMDB_PURE=1.
    patching file lmdb.h
    patching file mdb.c
    py-lmdb: Using CPython extension; override with LMDB_FORCE_CFFI=1.
    running install
    running build
    running build_py
    creating build/lib.linux-x86_64-2.7
    creating build/lib.linux-x86_64-2.7/lmdb
    copying lmdb/tool.py -> build/lib.linux-x86_64-2.7/lmdb
    copying lmdb/__init__.py -> build/lib.linux-x86_64-2.7/lmdb
    copying lmdb/_config.py -> build/lib.linux-x86_64-2.7/lmdb
    copying lmdb/__main__.py -> build/lib.linux-x86_64-2.7/lmdb
    copying lmdb/cffi.py -> build/lib.linux-x86_64-2.7/lmdb
    running build_ext
    building 'cpython' extension
    creating build/temp.linux-x86_64-2.7
    creating build/temp.linux-x86_64-2.7/lmdb
    creating build/temp.linux-x86_64-2.7/build
    creating build/temp.linux-x86_64-2.7/build/lib
    gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ilib/py-lmdb -Ibuild/lib -I/home/shachar/.pyenv/versions/2.7.18/include/python2.7 -c lmdb/cpython.c -o build/temp.linux-x86_64-2.7/lmdb/cpython.o -DHAVE_PATCHED_LMDB=1 -UNDEBUG -w
    lmdb/cpython.c:3819:15: error: variable ‘moduledef’ has initializer but incomplete type
     3819 | static struct PyModuleDef moduledef = {
          |               ^~~~~~~~~~~
    lmdb/cpython.c:3820:5: error: ‘PyModuleDef_HEAD_INIT’ undeclared here (not in a function)
     3820 |     PyModuleDef_HEAD_INIT,
          |     ^~~~~~~~~~~~~~~~~~~~~
    lmdb/cpython.c:3819:27: error: storage size of ‘moduledef’ isn’t known
     3819 | static struct PyModuleDef moduledef = {
          |                           ^~~~~~~~~
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/shachar/.pyenv/versions/2.7.18/bin/python2.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-kGq0cd/lmdb/setup.py'"'"'; __file__='"'"'/tmp/pip-install-kGq0cd/lmdb/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-zQBrCU/install-record.txt --single-version-externally-managed --compile --install-headers /home/shachar/.pyenv/versions/2.7.18/include/python2.7/lmdb Check the logs for full command output.

also with wheel installed - lmdb fails to install:

$ pip install lmdb --upgrade
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Collecting lmdb
  Using cached lmdb-1.5.1.tar.gz (881 kB)
Building wheels for collected packages: lmdb
  Building wheel for lmdb (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/shachar/.pyenv/versions/2.7.18/bin/python2.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-0zfJQS/lmdb/setup.py'"'"'; __file__='"'"'/tmp/pip-install-0zfJQS/lmdb/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-0qAs_a
       cwd: /tmp/pip-install-0zfJQS/lmdb/
  Complete output (31 lines):
  py-lmdb: Using bundled liblmdb with py-lmdb patches; override with LMDB_FORCE_SYSTEM=1 or LMDB_PURE=1.
  patching file lmdb.h
  patching file mdb.c
  py-lmdb: Using CPython extension; override with LMDB_FORCE_CFFI=1.
  running bdist_wheel
  running build
  running build_py
  creating build/lib.linux-x86_64-2.7
  creating build/lib.linux-x86_64-2.7/lmdb
  copying lmdb/tool.py -> build/lib.linux-x86_64-2.7/lmdb
  copying lmdb/__init__.py -> build/lib.linux-x86_64-2.7/lmdb
  copying lmdb/_config.py -> build/lib.linux-x86_64-2.7/lmdb
  copying lmdb/__main__.py -> build/lib.linux-x86_64-2.7/lmdb
  copying lmdb/cffi.py -> build/lib.linux-x86_64-2.7/lmdb
  running build_ext
  building 'cpython' extension
  creating build/temp.linux-x86_64-2.7
  creating build/temp.linux-x86_64-2.7/lmdb
  creating build/temp.linux-x86_64-2.7/build
  creating build/temp.linux-x86_64-2.7/build/lib
  gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ilib/py-lmdb -Ibuild/lib -I/home/shachar/.pyenv/versions/2.7.18/include/python2.7 -c lmdb/cpython.c -o build/temp.linux-x86_64-2.7/lmdb/cpython.o -DHAVE_PATCHED_LMDB=1 -UNDEBUG -w
  lmdb/cpython.c:3819:15: error: variable ‘moduledef’ has initializer but incomplete type
   3819 | static struct PyModuleDef moduledef = {
        |               ^~~~~~~~~~~
  lmdb/cpython.c:3820:5: error: ‘PyModuleDef_HEAD_INIT’ undeclared here (not in a function)
   3820 |     PyModuleDef_HEAD_INIT,
        |     ^~~~~~~~~~~~~~~~~~~~~
  lmdb/cpython.c:3819:27: error: storage size of ‘moduledef’ isn’t known
   3819 | static struct PyModuleDef moduledef = {
        |                           ^~~~~~~~~
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for lmdb
  Running setup.py clean for lmdb
Failed to build lmdb
Installing collected packages: lmdb
  Attempting uninstall: lmdb
    Found existing installation: lmdb 1.4.1
    Uninstalling lmdb-1.4.1:
      Successfully uninstalled lmdb-1.4.1
    Running setup.py install for lmdb ... error
    ERROR: Command errored out with exit status 1:
     command: /home/shachar/.pyenv/versions/2.7.18/bin/python2.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-0zfJQS/lmdb/setup.py'"'"'; __file__='"'"'/tmp/pip-install-0zfJQS/lmdb/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-TfmMZX/install-record.txt --single-version-externally-managed --compile --install-headers /home/shachar/.pyenv/versions/2.7.18/include/python2.7/lmdb
         cwd: /tmp/pip-install-0zfJQS/lmdb/
    Complete output (31 lines):
    py-lmdb: Using bundled liblmdb with py-lmdb patches; override with LMDB_FORCE_SYSTEM=1 or LMDB_PURE=1.
    patching file lmdb.h
    patching file mdb.c
    py-lmdb: Using CPython extension; override with LMDB_FORCE_CFFI=1.
    running install
    running build
    running build_py
    creating build/lib.linux-x86_64-2.7
    creating build/lib.linux-x86_64-2.7/lmdb
    copying lmdb/tool.py -> build/lib.linux-x86_64-2.7/lmdb
    copying lmdb/__init__.py -> build/lib.linux-x86_64-2.7/lmdb
    copying lmdb/_config.py -> build/lib.linux-x86_64-2.7/lmdb
    copying lmdb/__main__.py -> build/lib.linux-x86_64-2.7/lmdb
    copying lmdb/cffi.py -> build/lib.linux-x86_64-2.7/lmdb
    running build_ext
    building 'cpython' extension
    creating build/temp.linux-x86_64-2.7
    creating build/temp.linux-x86_64-2.7/lmdb
    creating build/temp.linux-x86_64-2.7/build
    creating build/temp.linux-x86_64-2.7/build/lib
    gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ilib/py-lmdb -Ibuild/lib -I/home/shachar/.pyenv/versions/2.7.18/include/python2.7 -c lmdb/cpython.c -o build/temp.linux-x86_64-2.7/lmdb/cpython.o -DHAVE_PATCHED_LMDB=1 -UNDEBUG -w
    lmdb/cpython.c:3819:15: error: variable ‘moduledef’ has initializer but incomplete type
     3819 | static struct PyModuleDef moduledef = {
          |               ^~~~~~~~~~~
    lmdb/cpython.c:3820:5: error: ‘PyModuleDef_HEAD_INIT’ undeclared here (not in a function)
     3820 |     PyModuleDef_HEAD_INIT,
          |     ^~~~~~~~~~~~~~~~~~~~~
    lmdb/cpython.c:3819:27: error: storage size of ‘moduledef’ isn’t known
     3819 | static struct PyModuleDef moduledef = {
          |                           ^~~~~~~~~
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
  Rolling back uninstall of lmdb
  Moving to /home/shachar/.pyenv/versions/2.7.18/lib/python2.7/site-packages/lmdb-1.4.1-py2.7.egg-info
   from /home/shachar/.pyenv/versions/2.7.18/lib/python2.7/site-packages/~mdb-1.4.1-py2.7.egg-info
  Moving to /home/shachar/.pyenv/versions/2.7.18/lib/python2.7/site-packages/lmdb/
   from /home/shachar/.pyenv/versions/2.7.18/lib/python2.7/site-packages/~mdb
ERROR: Command errored out with exit status 1: /home/shachar/.pyenv/versions/2.7.18/bin/python2.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-0zfJQS/lmdb/setup.py'"'"'; __file__='"'"'/tmp/pip-install-0zfJQS/lmdb/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-TfmMZX/install-record.txt --single-version-externally-managed --compile --install-headers /home/shachar/.pyenv/versions/2.7.18/include/python2.7/lmdb Check the logs for full command output.

but v 1.4.1 works well as before - 

$ pip install lmdb==1.4.1
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Collecting lmdb==1.4.1
  Downloading lmdb-1.4.1.tar.gz (881 kB)
     |████████████████████████████████| 881 kB 20.4 MB/s 
Using legacy 'setup.py install' for lmdb, since package 'wheel' is not installed.
Installing collected packages: lmdb
    Running setup.py install for lmdb ... done
Successfully installed lmdb-1.4.1

Describe What You Expected To Happen

I expected installation to succeed as it did with v 1.4.1 and before :)

Describe What Happened Instead

Installation failed, reverted back to last known version

vEpiphyte commented 4 months ago

@chkp-shacharah py-lmdb removed support for Python 2.7. See the following changelog entries:

2024-06-30 1.5.0
* Add Python 3.12 binaries.

* Update bundled LMDB to 0.9.31.

* Remove Python 2.7 support.

You would need to use an earlier version if you're working with Python 2.7; or update your Python interpreter.

jnwatson commented 4 months ago

It is also mentioned right on the README and on the front of the github page.

I'm surprised that pip tried to install it though. The metadata on the package is correct: https://pypi.org/project/lmdb/

Keeping this open to remind me to put in a more obvious error out if Python is too old.

jnwatson commented 4 months ago

I see the problem with the package metadata. A correct example is here: https://pypi.org/project/numpy/. On the left bar, you can see a "Requires: Python >=3.9". I need something like that and I believe pip will refuse to install. This is fantastic because I'm going to need to bump the version again and remove 1.5.0 and 1.5.1 from pypi.