Closed emabrey closed 1 year ago
ikelos
commented
2 years ago This seems like an issue with pip install python-snappy? We define the requirement, which should in turn define any requirements it has. Please provide the failure message, since I don't believe this is a volatility issue, but rather a python-snappy issue.
emabrey
commented
2 years ago It isn't a python-snappy issue. Installing the C library via the system package manager before the call to pip is the best way to do this. Building the library from source and installing it via pip is a nightmare that is completely avoided by using the pre-existing pre-built packages from the system package manager.
The issue is simply that the installation instructions for python-snappy tell you to install the C libraries first, as they are transitive dependencies. Without them, python-snappy fails with this error, which is caused because it can't find the header files which are installed alongside the devel libraries. Using the pre-built version of 0.5.3 that comes with Kali is seamless, as that means you don't have to build from source so you don't need the devel headers, but having to build for python-snappy 0.6.0 means that unless you have the C library installed first, you won't install python-snappy:0.6.0.
What the error looks like:
──(emabrey㉿2603-6010-c000-29e4-0000-0000-0000-1227)-[~/volatility3]
└─$ pip3 list | grep snappy
python-snappy 0.5.3
┌──(emabrey㉿2603-6010-c000-29e4-0000-0000-0000-1227)-[~/volatility3]
└─$ pip3 install -r requirements.txt
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: pefile>=2017.8.1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 2)) (2022.5.30)
Requirement already satisfied: yara-python>=3.8.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 8)) (4.2.0)
Requirement already satisfied: capstone>=3.0.5 in /home/emabrey/.local/lib/python3.10/site-packages (from -r requirements.txt (line 12)) (5.0.0rc2)
Requirement already satisfied: pycryptodome in /home/emabrey/.local/lib/python3.10/site-packages (from -r requirements.txt (line 15)) (3.15.0)
Requirement already satisfied: jsonschema>=2.3.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 18)) (3.2.0)
Requirement already satisfied: leechcorepyc>=2.4.0 in /home/emabrey/.local/lib/python3.10/site-packages (from -r requirements.txt (line 21)) (2.11.1)
Collecting python-snappy==0.6.0
Downloading python-snappy-0.6.0.tar.gz (21 kB)
Preparing metadata (setup.py) ... done
Building wheels for collected packages: python-snappy
Building wheel for python-snappy (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [24 lines of output]
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.10
creating build/lib.linux-x86_64-3.10/snappy
copying snappy/snappy.py -> build/lib.linux-x86_64-3.10/snappy
copying snappy/__main__.py -> build/lib.linux-x86_64-3.10/snappy
copying snappy/snappy_cffi.py -> build/lib.linux-x86_64-3.10/snappy
copying snappy/hadoop_snappy.py -> build/lib.linux-x86_64-3.10/snappy
copying snappy/__init__.py -> build/lib.linux-x86_64-3.10/snappy
copying snappy/snappy_cffi_builder.py -> build/lib.linux-x86_64-3.10/snappy
copying snappy/snappy_formats.py -> build/lib.linux-x86_64-3.10/snappy
running build_ext
building 'snappy._snappy' extension
creating build/temp.linux-x86_64-3.10
creating build/temp.linux-x86_64-3.10/snappy
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.10 -c snappy/crc32c.c -o build/temp.linux-x86_64-3.10/snappy/crc32c.o
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.10 -c snappy/snappymodule.cc -o build/temp.linux-x86_64-3.10/snappy/snappymodule.o
snappy/snappymodule.cc:32:10: fatal error: snappy-c.h: No such file or directory
32 | #include <snappy-c.h>
| ^~~~~~~~~~~~
compilation terminated.
error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for python-snappy
Running setup.py clean for python-snappy
Failed to build python-snappy
Installing collected packages: python-snappy
Running setup.py install for python-snappy ... error
error: subprocess-exited-with-error
× Running setup.py install for python-snappy did not run successfully.
│ exit code: 1
╰─> [26 lines of output]
running install
/usr/lib/python3/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.10
creating build/lib.linux-x86_64-3.10/snappy
copying snappy/snappy.py -> build/lib.linux-x86_64-3.10/snappy
copying snappy/__main__.py -> build/lib.linux-x86_64-3.10/snappy
copying snappy/snappy_cffi.py -> build/lib.linux-x86_64-3.10/snappy
copying snappy/hadoop_snappy.py -> build/lib.linux-x86_64-3.10/snappy
copying snappy/__init__.py -> build/lib.linux-x86_64-3.10/snappy
copying snappy/snappy_cffi_builder.py -> build/lib.linux-x86_64-3.10/snappy
copying snappy/snappy_formats.py -> build/lib.linux-x86_64-3.10/snappy
running build_ext
building 'snappy._snappy' extension
creating build/temp.linux-x86_64-3.10
creating build/temp.linux-x86_64-3.10/snappy
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.10 -c snappy/crc32c.c -o build/temp.linux-x86_64-3.10/snappy/crc32c.o
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.10 -c snappy/snappymodule.cc -o build/temp.linux-x86_64-3.10/snappy/snappymodule.o
snappy/snappymodule.cc:32:10: fatal error: snappy-c.h: No such file or directory
32 | #include <snappy-c.h>
| ^~~~~~~~~~~~
compilation terminated.
error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure
× Encountered error while trying to install package.
╰─> python-snappy
note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
┌──(emabrey㉿2603-6010-c000-29e4-0000-0000-0000-1227)-[~/volatility3]
└─$ What it looks like if you install the C library first:
┌──(emabrey㉿2603-6010-c000-29e4-0000-0000-0000-1227)-[~/volatility3]
└─$ pip3 list | grep snappy
python-snappy 0.5.3
┌──(emabrey㉿2603-6010-c000-29e4-0000-0000-0000-1227)-[~/volatility3]
└─$ sudo apt-get install libsnappy-dev
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
apt-file docutils-common espeak-ng-data fonts-linuxlibertine gir1.2-atspi-2.0 gir1.2-ayatanaappindicator3-0.1 gir1.2-gstreamer-1.0 gir1.2-nm-1.0 gir1.2-wnck-3.0 gir1.2-xfconf-0 icu-devtools
libao-common libao4 libapt-pkg-perl libatk-adaptor libbrlapi0.8 libclang-cpp11 libdotconf0 libespeak-ng1 libexporter-tiny-perl libgarcon-gtk3-1-0 libicu-dev libkeybinder-3.0-0
liblist-moreutils-perl liblist-moreutils-xs-perl liblouis-data liblouis20 libmspack0 libncurses-dev libpcaudio0 libregexp-assemble-perl libsonic0 libspeechd2 libtinfo-dev libxdo3 libxfce4ui-utils
libxml2-dev libxpresent1 libz3-dev light-locker llvm-13 llvm-13-runtime perl-tk python-apt-common python-pkg-resources python-setuptools sound-icons speech-dispatcher
speech-dispatcher-audio-plugins speech-dispatcher-espeak-ng tango-icon-theme x11-session-utils xbrlapi xdotool xfce4-appfinder xfce4-panel xfce4-pulseaudio-plugin xfdesktop4 xfdesktop4-data xfwm4
xiccd xinit xkbset xml-core xorg zerofree
Use 'sudo apt autoremove' to remove them.
The following NEW packages will be installed:
libsnappy-dev
0 upgraded, 1 newly installed, 0 to remove and 9 not upgraded.
Need to get 0 B/40.1 kB of archives.
After this operation, 145 kB of additional disk space will be used.
Selecting previously unselected package libsnappy-dev:amd64.
(Reading database ... 325860 files and directories currently installed.)
Preparing to unpack .../libsnappy-dev_1.1.9-2_amd64.deb ...
Unpacking libsnappy-dev:amd64 (1.1.9-2) ...
Setting up libsnappy-dev:amd64 (1.1.9-2) ...
┌──(emabrey㉿2603-6010-c000-29e4-0000-0000-0000-1227)-[~/volatility3]
└─$ pip3 list | grep snappy
python-snappy 0.5.3
┌──(emabrey㉿2603-6010-c000-29e4-0000-0000-0000-1227)-[~/volatility3]
└─$ pip3 install -r requirements.txt
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: pefile>=2017.8.1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 2)) (2022.5.30)
Requirement already satisfied: yara-python>=3.8.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 8)) (4.2.0)
Requirement already satisfied: capstone>=3.0.5 in /home/emabrey/.local/lib/python3.10/site-packages (from -r requirements.txt (line 12)) (5.0.0rc2)
Requirement already satisfied: pycryptodome in /home/emabrey/.local/lib/python3.10/site-packages (from -r requirements.txt (line 15)) (3.15.0)
Requirement already satisfied: jsonschema>=2.3.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 18)) (3.2.0)
Requirement already satisfied: leechcorepyc>=2.4.0 in /home/emabrey/.local/lib/python3.10/site-packages (from -r requirements.txt (line 21)) (2.11.1)
Collecting python-snappy==0.6.0
Using cached python-snappy-0.6.0.tar.gz (21 kB)
Preparing metadata (setup.py) ... done
Building wheels for collected packages: python-snappy
Building wheel for python-snappy (setup.py) ... done
Created wheel for python-snappy: filename=python_snappy-0.6.0-cp310-cp310-linux_x86_64.whl size=37142 sha256=dbd8c6f151e070861b114463b6b81d8eb42564e1fc60332f8e59757e6ce4e376
Stored in directory: /home/emabrey/.cache/pip/wheels/5d/cb/72/ac8185dbbfbdfabbe6d1bc02e27884884bedf1af19365d94f4
Successfully built python-snappy
Installing collected packages: python-snappy
Successfully installed python-snappy-0.6.0
┌──(emabrey㉿2603-6010-c000-29e4-0000-0000-0000-1227)-[~/volatility3]
└─$ pip3 list | grep snappy
python-snappy 0.6.0
┌──(emabrey㉿2603-6010-c000-29e4-0000-0000-0000-1227)-[~/volatility3]
└─$
Is your feature request related to a problem? Please describe. The README does not instruct users to install the Snappy c library for an installation using
requirements.txt, and failing to have that library installed will cause the commandpip3 install -r requirements.txtto fail with an error.Describe the solution you'd like Add instructions to the README to install the Snappy C library. For example:
sudo apt-get install libsnappy-devorsudo yum install libsnappy-develDescribe alternatives you've considered It may be possible to create a script which does this automatically, and have the user call that instead of pip directly, but that would be an unnecessary complexity.
Additional information The installation fails with a failure to find
snappy-c.h.