Open parsa97 opened 1 year ago
I couldn't get 'pip install python-rocksdb' to work on Linux Mint (Debian based) x86_64. Tried Luke's Dockerfile - same failure. Hopes were high with your Dockerfile - same failure. I don't know what's going on or while it's failing - feels like I've tried every possible dependency and combination.
Without docker, here's what I have, yours looks similar:
sudo apt-get remove -y libdpkg-perl #my machine had a newer version, preventing build-essential to be installed sudo apt-get install -y --fix-broken --no-install-recommends gcc g++ fakeroot python3 python3-pip python3-dev \ build-essential librocksdb-dev libsnappy-dev zlib1g-dev libbz2-dev libgflags-dev liblz4-dev libzstd-dev \ rocksdb-tools librocksdb-6.11 sudo python3 -m pip install --upgrade pip
pip install python-rocksdb
pip install.[rocksdb]
$ pip3 install python-rocksdb >log 2>&1 && tail -n100 log
Defaulting to user installation because normal site-packages is not writeable Collecting python-rocksdb Using cached python-rocksdb-0.7.0.tar.gz (219 kB) Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Requirement already satisfied: setuptools>=25 in /home/parman/.local/lib/python3.10/site-packages (from python-rocksdb) (69.0.2) Building wheels for collected packages: python-rocksdb Building wheel for python-rocksdb (setup.py): started Building wheel for python-rocksdb (setup.py): finished with status 'error' error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully. │ exit code: 1 ╰─> [7965 lines of output] /home/parman/.local/lib/python3.10/site-packages/setuptools/init.py:80: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated. !!
********************************************************************************
Requirements should be satisfied by a PEP 517 installer.
If you are using pip, you can try `pip install --use-pep517`.
********************************************************************************
!!
dist.fetch_build_eggs(dist.setup_requires)
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-310
creating build/lib.linux-x86_64-cpython-310/rocksdb
copying rocksdb/errors.py -> build/lib.linux-x86_64-cpython-310/rocksdb
copying rocksdb/__init__.py -> build/lib.linux-x86_64-cpython-310/rocksdb
copying rocksdb/merge_operators.py -> build/lib.linux-x86_64-cpython-310/rocksdb
copying rocksdb/interfaces.py -> build/lib.linux-x86_64-cpython-310/rocksdb
creating build/lib.linux-x86_64-cpython-310/rocksdb/tests
copying rocksdb/tests/test_memtable.py -> build/lib.linux-x86_64-cpython-310/rocksdb/tests
copying rocksdb/tests/test_options.py -> build/lib.linux-x86_64-cpython-310/rocksdb/tests
copying rocksdb/tests/test_db.py -> build/lib.linux-x86_64-cpython-310/rocksdb/tests
copying rocksdb/tests/__init__.py -> build/lib.linux-x86_64-cpython-310/rocksdb/tests
running egg_info
writing python_rocksdb.egg-info/PKG-INFO
writing dependency_links to python_rocksdb.egg-info/dependency_links.txt
writing requirements to python_rocksdb.egg-info/requires.txt
writing top-level names to python_rocksdb.egg-info/top_level.txt
reading manifest file 'python_rocksdb.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'python_rocksdb.egg-info/SOURCES.txt'
/home/parman/.local/lib/python3.10/site-packages/setuptools/command/build_py.py:207: _Warning: Package 'rocksdb.cpp' is absent from the `packages` configuration.
!!
********************************************************************************
############################
# Package would be ignored #
############################
Python recognizes 'rocksdb.cpp' as an importable package[^1],
but it is absent from setuptools' `packages` configuration.
This leads to an ambiguous overall configuration. If you want to distribute this
package, please make sure that 'rocksdb.cpp' is explicitly added
to the `packages` configuration field.
Alternatively, you can also rely on setuptools' discovery methods
(for example by using `find_namespace_packages(...)`/`find_namespace:`
instead of `find_packages(...)`/`find:`).
You can read more about "package discovery" on setuptools documentation page:
- https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
If you don't want 'rocksdb.cpp' to be distributed and are
already explicitly excluding 'rocksdb.cpp' via
`find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
you can try to use `exclude_package_data`, or `include-package-data=False` in
combination with a more fine grained `package-data` configuration.
You can read more about "package data files" on setuptools documentation page:
- https://setuptools.pypa.io/en/latest/userguide/datafiles.html
[^1]: For Python, any directory (with suitable naming) can be imported,
even if it does not contain any `.py` files.
On the other hand, currently there is no concept of package data
directory, all directories are treated like packages.
********************************************************************************
!!
check.warn(importable)
copying rocksdb/_rocksdb.cpp -> build/lib.linux-x86_64-cpython-310/rocksdb
copying rocksdb/_rocksdb.pyx -> build/lib.linux-x86_64-cpython-310/rocksdb
copying rocksdb/backup.pxd -> build/lib.linux-x86_64-cpython-310/rocksdb
copying rocksdb/cache.pxd -> build/lib.linux-x86_64-cpython-310/rocksdb
copying rocksdb/comparator.pxd -> build/lib.linux-x86_64-cpython-310/rocksdb
copying rocksdb/db.pxd -> build/lib.linux-x86_64-cpython-310/rocksdb
copying rocksdb/env.pxd -> build/lib.linux-x86_64-cpython-310/rocksdb
copying rocksdb/filter_policy.pxd -> build/lib.linux-x86_64-cpython-310/rocksdb
copying rocksdb/iterator.pxd -> build/lib.linux-x86_64-cpython-310/rocksdb
copying rocksdb/logger.pxd -> build/lib.linux-x86_64-cpython-310/rocksdb
copying rocksdb/memtablerep.pxd -> build/lib.linux-x86_64-cpython-310/rocksdb
copying rocksdb/merge_operator.pxd -> build/lib.linux-x86_64-cpython-310/rocksdb
$ tail -n100 log rocksdb/_rocksdb.pyx:2413:37: Calling gil-requiring function not allowed without gil
Error compiling Cython file:
------------------------------------------------------------
...
def get_backup_info(self):
cdef vector[backup.BackupInfo] backup_info
with nogil:
self.engine.GetBackupInfo(cython.address(backup_info))
^
------------------------------------------------------------
rocksdb/_rocksdb.pyx:2413:23: Accessing Python attribute not allowed without gil
Error compiling Cython file:
------------------------------------------------------------
...
def get_backup_info(self):
cdef vector[backup.BackupInfo] backup_info
with nogil:
self.engine.GetBackupInfo(cython.address(backup_info))
^
------------------------------------------------------------
rocksdb/_rocksdb.pyx:2413:37: Constructing Python tuple not allowed without gil
Error compiling Cython file:
------------------------------------------------------------
...
def get_backup_info(self):
cdef vector[backup.BackupInfo] backup_info
with nogil:
self.engine.GetBackupInfo(cython.address(backup_info))
^
------------------------------------------------------------
rocksdb/_rocksdb.pyx:2413:44: Converting to Python object not allowed without gil
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "/tmp/pip-install-txclzhkh/python-rocksdb_bf4eb2ade9b74c4e90c37c2134ed3948/setup.py", line 21, in <module>
setup(
File "/home/parman/.local/lib/python3.10/site-packages/setuptools/__init__.py", line 103, in setup
return distutils.core.setup(**attrs)
File "/home/parman/.local/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
return run_commands(dist)
File "/home/parman/.local/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
dist.run_commands()
File "/home/parman/.local/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
self.run_command(cmd)
File "/home/parman/.local/lib/python3.10/site-packages/setuptools/dist.py", line 963, in run_command
super().run_command(command)
File "/home/parman/.local/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/usr/lib/python3/dist-packages/wheel/bdist_wheel.py", line 299, in run
self.run_command('build')
File "/home/parman/.local/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "/home/parman/.local/lib/python3.10/site-packages/setuptools/dist.py", line 963, in run_command
super().run_command(command)
File "/home/parman/.local/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/home/parman/.local/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 131, in run
self.run_command(cmd_name)
File "/home/parman/.local/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "/home/parman/.local/lib/python3.10/site-packages/setuptools/dist.py", line 963, in run_command
super().run_command(command)
File "/home/parman/.local/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/home/parman/.local/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 88, in run
_build_ext.run(self)
File "/home/parman/.local/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
self.build_extensions()
File "/home/parman/.local/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
self._build_extensions_serial()
File "/home/parman/.local/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
self.build_extension(ext)
File "/home/parman/.local/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 249, in build_extension
_build_ext.build_extension(self, ext)
File "/tmp/pip-install-txclzhkh/python-rocksdb_bf4eb2ade9b74c4e90c37c2134ed3948/.eggs/Cython-3.0.8-py3.10-linux-x86_64.egg/Cython/Distutils/build_ext.py", line 130, in build_extension
new_ext = cythonize(
File "/tmp/pip-install-txclzhkh/python-rocksdb_bf4eb2ade9b74c4e90c37c2134ed3948/.eggs/Cython-3.0.8-py3.10-linux-x86_64.egg/Cython/Build/Dependencies.py", line 1154, in cythonize
cythonize_one(*args)
File "/tmp/pip-install-txclzhkh/python-rocksdb_bf4eb2ade9b74c4e90c37c2134ed3948/.eggs/Cython-3.0.8-py3.10-linux-x86_64.egg/Cython/Build/Dependencies.py", line 1321, in cythonize_one
raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: rocksdb/_rocksdb.pyx
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for python-rocksdb Running setup.py clean for python-rocksdb Failed to build python-rocksdb ERROR: Could not build wheels for python-rocksdb, which is required to install pyproject.toml-based projects parman@parman:~/parmanode/electrumx$
@ArmanTheParman Thank you for reporting this issue, I made some changes please test it
Great, I will! Thank you
I tested it on a mac, x86_64
% docker build -t test_rocks .
[+] Building 49.0s (7/8)
=> [internal] load .dockerignore 0.4s
=> => transferring context: 2B 0.0s
=> [internal] load build definition from Dockerfile 0.8s
=> => transferring dockerfile: 1.06kB 0.0s
=> [internal] load metadata for docker.io/library/python:3.7-alpine3.11 23.0s
=> [internal] load build context 0.1s
=> => transferring context: 308B 0.0s
=> [1/4] FROM docker.io/library/python:3.7-alpine3.11@sha256:0ddccce8820c89f39cde6c482b3ba8bcad9e6652a060c329b7b963112df0b8e7 5.2s
=> => resolve docker.io/library/python:3.7-alpine3.11@sha256:0ddccce8820c89f39cde6c482b3ba8bcad9e6652a060c329b7b963112df0b8e7 0.1s
=> => sha256:233d96b408279e8bd3bc2420e6b070d4af559b50d792ee416508669ba88d68f2 299.55kB / 299.55kB 0.9s
=> => sha256:64ad073959edc68f72e9a6d32cb8f9439f19c45eb5c736466860ed166465d3d7 10.72MB / 10.72MB 2.8s
=> => sha256:0ddccce8820c89f39cde6c482b3ba8bcad9e6652a060c329b7b963112df0b8e7 1.65kB / 1.65kB 0.0s
=> => sha256:1724b17cbf37548616325811484dd5a60351ab06bca4c5367b5c297c5e193e01 1.37kB / 1.37kB 0.0s
=> => sha256:8b32838a7ca667115c351dff8279f5a4bf35c20cb7bb98609302e87a92b5cf18 7.83kB / 7.83kB 0.0s
=> => sha256:0a6724ff3fcd51338afdfdc2b1d4ffd04569818e31efad957213d67c29b45101 2.81MB / 2.81MB 1.3s
=> => sha256:6d21bc3309f3343f3acab6ddd607a668dc5f90ef0e79bf9ffb6ba875708ee440 230B / 230B 1.6s
=> => extracting sha256:0a6724ff3fcd51338afdfdc2b1d4ffd04569818e31efad957213d67c29b45101 0.2s
=> => sha256:7367ec39bfc942f89263045daa5250cbbb9b9e26c2184d119aa5f2e1de0300f4 2.16MB / 2.16MB 2.5s
=> => extracting sha256:233d96b408279e8bd3bc2420e6b070d4af559b50d792ee416508669ba88d68f2 0.1s
=> => extracting sha256:64ad073959edc68f72e9a6d32cb8f9439f19c45eb5c736466860ed166465d3d7 1.1s
=> => extracting sha256:6d21bc3309f3343f3acab6ddd607a668dc5f90ef0e79bf9ffb6ba875708ee440 0.0s
=> => extracting sha256:7367ec39bfc942f89263045daa5250cbbb9b9e26c2184d119aa5f2e1de0300f4 0.5s
=> [2/4] COPY ./bin /usr/local/bin 0.7s
=> ERROR [3/4] RUN chmod a+x /usr/local/bin/* && apk add --no-cache git build-base openssl && apk add --no-cache --reposito 18.7s
[3/4] RUN chmod a+x /usr/local/bin/ && apk add --no-cache git build-base openssl && apk add --no-cache --repository http://dl-cdn.alpinelinux.org/alpine/v3.11/main leveldb-dev && apk add --no-cache --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing rocksdb-dev && pip install aiohttp pylru plyvel websockets python-rocksdb uvloop && git clone -b 1.16.0 https://github.com/spesmilo/electrumx.git && cd electrumx && python setup.py install && apk del git build-base && rm -rf /tmp/:
0 1.120 fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
0 2.105 fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
0 3.083 (1/23) Installing libgcc (9.3.0-r0)
0 3.095 (2/23) Installing libstdc++ (9.3.0-r0)
0 3.180 (3/23) Installing binutils (2.33.1-r1)
0 3.555 (4/23) Installing libmagic (5.37-r1)
0 3.616 (5/23) Installing file (5.37-r1)
0 3.632 (6/23) Installing gmp (6.1.2-r1)
0 3.675 (7/23) Installing isl (0.18-r0)
0 3.774 (8/23) Installing libgomp (9.3.0-r0)
0 3.795 (9/23) Installing libatomic (9.3.0-r0)
0 3.814 (10/23) Installing mpfr4 (4.0.2-r1)
0 3.857 (11/23) Installing mpc1 (1.1.0-r1)
0 3.873 (12/23) Installing gcc (9.3.0-r0)
0 8.557 (13/23) Installing musl-dev (1.1.24-r3)
0 10.74 (14/23) Installing libc-dev (0.7.2-r0)
0 10.75 (15/23) Installing g++ (9.3.0-r0)
0 13.64 (16/23) Installing make (4.2.1-r2)
0 13.67 (17/23) Installing fortify-headers (1.1-r0)
0 13.68 (18/23) Installing build-base (0.5-r1)
0 13.70 (19/23) Installing nghttp2-libs (1.40.0-r1)
0 13.72 (20/23) Installing libcurl (7.79.1-r0)
0 13.76 (21/23) Installing pcre2 (10.34-r1)
0 13.81 (22/23) Installing git (2.24.4-r0)
0 14.90 (23/23) Installing openssl (1.1.1l-r0)
0 14.95 Executing busybox-1.31.1-r9.trigger
0 14.96 OK: 199 MiB in 57 packages
0 14.99 fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
0 15.40 fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
0 15.86 (1/3) Installing snappy (1.1.7-r1)
0 15.91 (2/3) Installing leveldb (1.22-r0)
0 15.94 (3/3) Installing leveldb-dev (1.22-r0)
0 15.96 OK: 199 MiB in 60 packages
0 15.99 fetch http://dl-cdn.alpinelinux.org/alpine/edge/testing/x86_64/APKINDEX.tar.gz
0 16.83 fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
0 16.83 WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/edge/testing/x86_64/APKINDEX.tar.gz: UNTRUSTED signature
0 17.04 fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
0 17.27 ERROR: unsatisfiable constraints:
0 17.29 rocksdb-dev (missing):
0 17.29 required by: world[rocksdb-dev]
Dockerfile:10
9 |
10 | >>> RUN chmod a+x /usr/local/bin/ && \ 11 | >>> apk add --no-cache git build-base openssl && \ 12 | >>> apk add --no-cache --repository http://dl-cdn.alpinelinux.org/alpine/v3.11/main leveldb-dev && \ 13 | >>> apk add --no-cache --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing rocksdb-dev && \ 14 | >>> pip install aiohttp pylru plyvel websockets python-rocksdb uvloop && \ 15 | >>> git clone -b $VERSION https://github.com/spesmilo/electrumx.git && \ 16 | >>> cd electrumx && \ 17 | >>> python setup.py install && \ 18 | >>> apk del git build-base && \ 19 | >>> rm -rf /tmp/ 20 |ERROR: failed to solve: process "/bin/sh -c chmod a+x /usr/local/bin/ && apk add --no-cache git build-base openssl && apk add --no-cache --repository http://dl-cdn.alpinelinux.org/alpine/v3.11/main leveldb-dev && apk add --no-cache --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing rocksdb-dev && pip install aiohttp pylru plyvel websockets python-rocksdb uvloop && git clone -b $VERSION https://github.com/spesmilo/electrumx.git && cd electrumx && python setup.py install && apk del git build-base && rm -rf /tmp/" did not complete successfully: exit code: 1
tested on Pi4, 8Gb
[3/4] RUN chmod a+x /usr/local/bin/ && apk add --no-cache git build-base openssl && apk add --no-cache --repository http://dl-cdn.alpinelinux.org/alpine/v3.11/main leveldb-dev && apk add --no-cache --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing rocksdb-dev && pip install aiohttp pylru plyvel websockets python-rocksdb uvloop && git clone -b 1.16.0 https://github.com/spesmilo/electrumx.git && cd electrumx && python setup.py install && apk del git build-base && rm -rf /tmp/: 13.85 (2/3) Installing leveldb (1.22-r0) 13.88 (3/3) Installing leveldb-dev (1.22-r0) 13.92 OK: 189 MiB in 60 packages 14.01 fetch http://dl-cdn.alpinelinux.org/alpine/edge/testing/aarch64/APKINDEX.tar.gz 16.83 WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/edge/testing/aarch64/APKINDEX.tar.gz: UNTRUSTED signature 16.84 fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/aarch64/APKINDEX.tar.gz 17.03 fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/aarch64/APKINDEX.tar.gz 17.39 ERROR: unsatisfiable constraints: 17.48 rocksdb-dev (missing): 17.48 required by: world[rocksdb-dev]
Dockerfile:10
9 |
10 | >>> RUN chmod a+x /usr/local/bin/ && \ 11 | >>> apk add --no-cache git build-base openssl && \ 12 | >>> apk add --no-cache --repository http://dl-cdn.alpinelinux.org/alpine/v3.11/main leveldb-dev && \ 13 | >>> apk add --no-cache --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing rocksdb-dev && \ 14 | >>> pip install aiohttp pylru plyvel websockets python-rocksdb uvloop && \ 15 | >>> git clone -b $VERSION https://github.com/spesmilo/electrumx.git && \ 16 | >>> cd electrumx && \ 17 | >>> python setup.py install && \ 18 | >>> apk del git build-base && \ 19 | >>> rm -rf /tmp/ 20 |ERROR: failed to solve: process "/bin/sh -c chmod a+x /usr/local/bin/ && apk add --no-cache git build-base openssl && apk add --no-cache --repository http://dl-cdn.alpinelinux.org/alpine/v3.11/main leveldb-dev && apk add --no-cache --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing rocksdb-dev && pip install aiohttp pylru plyvel websockets python-rocksdb uvloop && git clone -b $VERSION https://github.com/spesmilo/electrumx.git && cd electrumx && python setup.py install && apk del git build-base && rm -rf /tmp/" did not complete successfully: exit code: 1
I think you aren't in correct branch, because your logs contains alpine
and apk
and my docker file is based on debian.
Please checkout on this commit hash c4cce37ac2035f583992bdc1e81671746bfc5a78
I think you aren't in correct branch, because your logs contains
alpine
andapk
and my docker file is based on debian.Please checkout on this commit hash
c4cce37ac2035f583992bdc1e81671746bfc5a78
This report was for the master branch, but I'll test yours too.
There is some problem with the alpine and
rocksdb-dev
package, I replaced alpine with the Debian and split it into more layers with theRUN
to speedup builds.Also, I add some lines to README.md about the build.