search

volatilityfoundation / volatility3

Volatility 3.0 development
http://volatilityfoundation.org/
Other
2.42k stars 396 forks source link

Compiled executable did not contain needed dependencies because requirements did not install correctly #919

Closed lic-9 closed 8 months ago

lic-9 commented 1 year ago

Describe the bug I cannot run netstat plugin because it is not available in the list of plugins.

Context Volatility Version: Stable branch as of Friday, March 3rd, 14:40 UTC+1 Operating System: Ubuntu 20.04 PyInstaller version: 5.8.0

To Reproduce Steps to reproduce the behavior:

  1. git clone https://github.com/volatilityfoundation/volatility3.git
  2. cd volatility3/
  3. git checkout stable
  4. pip3 install -r requirements.txt
  5. Edit vol.spec as suggested here
  6. PYTHONPATH="." python3 -m PyInstaller vol.spec
  7. cd dist
  8. ./vol -vvv -h

Output of step 4 (pip3 install -r requirements.txt)

[ pikachu ] volatility3 (stable) $
    > pip3 install -r requirements.txt 
Defaulting to user installation because normal site-packages is not writeable
Collecting pefile>=2017.8.1
  Using cached pefile-2023.2.7-py3-none-any.whl (71 kB)
Requirement already satisfied: yara-python>=3.8.0 in /home/pikachu/.local/lib/python3.10/site-packages (from -r requirements.txt (line 8)) (4.2.3)
Collecting capstone>=3.0.5
  Using cached capstone-4.0.2-py2.py3-none-manylinux1_x86_64.whl (2.1 MB)
Requirement already satisfied: pycryptodome in /home/pikachu/.local/lib/python3.10/site-packages (from -r requirements.txt (line 15)) (3.17)
Collecting leechcorepyc>=2.4.0
  Using cached leechcorepyc-2.14.1-cp36-abi3-manylinux1_x86_64.whl (126 kB)
Collecting python-snappy==0.6.0
  Using cached python-snappy-0.6.0.tar.gz (21 kB)
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: python-snappy
  Building wheel for python-snappy (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [26 lines of output]
      /usr/lib/python3.10/distutils/dist.py:274: UserWarning: Unknown distribution option: 'cffi_modules'
        warnings.warn(msg)
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.10
      creating build/lib.linux-x86_64-3.10/snappy
      copying snappy/__main__.py -> build/lib.linux-x86_64-3.10/snappy
      copying snappy/__init__.py -> build/lib.linux-x86_64-3.10/snappy
      copying snappy/snappy_cffi.py -> build/lib.linux-x86_64-3.10/snappy
      copying snappy/snappy.py -> build/lib.linux-x86_64-3.10/snappy
      copying snappy/snappy_formats.py -> build/lib.linux-x86_64-3.10/snappy
      copying snappy/snappy_cffi_builder.py -> build/lib.linux-x86_64-3.10/snappy
      copying snappy/hadoop_snappy.py -> build/lib.linux-x86_64-3.10/snappy
      running build_ext
      building 'snappy._snappy' extension
      creating build/temp.linux-x86_64-3.10
      creating build/temp.linux-x86_64-3.10/snappy
      x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.10 -c snappy/crc32c.c -o build/temp.linux-x86_64-3.10/snappy/crc32c.o
      x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.10 -c snappy/snappymodule.cc -o build/temp.linux-x86_64-3.10/snappy/snappymodule.o
      snappy/snappymodule.cc:32:10: fatal error: snappy-c.h: No such file or directory
         32 | #include <snappy-c.h>
            |          ^~~~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for python-snappy
  Running setup.py clean for python-snappy
Failed to build python-snappy
Installing collected packages: python-snappy, pefile, leechcorepyc, capstone
  Running setup.py install for python-snappy ... error
  error: subprocess-exited-with-error

  × Running setup.py install for python-snappy did not run successfully.
  │ exit code: 1
  ╰─> [28 lines of output]
      /usr/lib/python3.10/distutils/dist.py:274: UserWarning: Unknown distribution option: 'cffi_modules'
        warnings.warn(msg)
      running install
      /usr/lib/python3/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.10
      creating build/lib.linux-x86_64-3.10/snappy
      copying snappy/__main__.py -> build/lib.linux-x86_64-3.10/snappy
      copying snappy/__init__.py -> build/lib.linux-x86_64-3.10/snappy
      copying snappy/snappy_cffi.py -> build/lib.linux-x86_64-3.10/snappy
      copying snappy/snappy.py -> build/lib.linux-x86_64-3.10/snappy
      copying snappy/snappy_formats.py -> build/lib.linux-x86_64-3.10/snappy
      copying snappy/snappy_cffi_builder.py -> build/lib.linux-x86_64-3.10/snappy
      copying snappy/hadoop_snappy.py -> build/lib.linux-x86_64-3.10/snappy
      running build_ext
      building 'snappy._snappy' extension
      creating build/temp.linux-x86_64-3.10
      creating build/temp.linux-x86_64-3.10/snappy
      x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.10 -c snappy/crc32c.c -o build/temp.linux-x86_64-3.10/snappy/crc32c.o
      x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.10 -c snappy/snappymodule.cc -o build/temp.linux-x86_64-3.10/snappy/snappymodule.o
      snappy/snappymodule.cc:32:10: fatal error: snappy-c.h: No such file or directory
         32 | #include <snappy-c.h>
            |          ^~~~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> python-snappy

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

Output of step 6 (PYTHONPATH="." python3 -m PyInstaller vol.spec)

[ pikachu ] volatility3 (stable) $
    > PYTHONPATH="." python3 -m PyInstaller vol.spec
145 INFO: PyInstaller: 5.8.0
145 INFO: Python: 3.10.6
147 INFO: Platform: Linux-5.15.0-43-generic-x86_64-with-glibc2.35
148 INFO: UPX is not available.
1842 INFO: Extending PYTHONPATH with paths
['/home/pikachu/Desktop/volatility3']
2029 INFO: checking Analysis
2052 INFO: Appending 'datas' from .spec
2065 INFO: checking PYZ
2069 WARNING: Ignoring icon; supported only on Windows and macOS!
2070 INFO: checking PKG
2072 INFO: Building because toc changed
2072 INFO: Building PKG (CArchive) vol.pkg
5904 INFO: Building PKG (CArchive) vol.pkg completed successfully.
5911 INFO: Bootloader /home/pikachu/.local/lib/python3.10/site-packages/PyInstaller/bootloader/Linux-64bit-intel/run
5911 INFO: checking EXE
5913 INFO: Rebuilding EXE-00.toc because vol missing
5913 INFO: Building EXE from EXE-00.toc
5913 INFO: Copying bootloader EXE to /home/pikachu/Desktop/volatility3/dist/vol
5913 INFO: Appending PKG archive to custom ELF section in EXE
5960 INFO: Building EXE from EXE-00.toc completed successfully.

Output of step 8 (./vol -vvv -h)

[ pikachu ] dist (stable) $
    > ./vol -vvv -h
Volatility 3 Framework 2.4.1
INFO     volatility3.cli: Volatility plugins path: ['/home/pikachu/Desktop/volatility3/dist/plugins', '/tmp/_MEIm9UJdo/volatility3/plugins', '/tmp/_MEIm9UJdo/volatility3/framework/plugins']
INFO     volatility3.cli: Volatility symbols path: ['/home/pikachu/Desktop/volatility3/dist/symbols', '/tmp/_MEIm9UJdo/volatility3/symbols', '/tmp/_MEIm9UJdo/volatility3/framework/symbols']
DEBUG    volatility3.framework: No module named 'pefile'
DEBUG    volatility3.framework: Failed to import module volatility3.plugins.windows.skeleton_key_check based on file: /tmp/_MEIm9UJdo/volatility3/framework/plugins/windows/skeleton_key_check.py
INFO     volatility3.plugins.windows.verinfo: Python pefile module not found, plugin (and dependent plugins) not available
DEBUG    volatility3.framework: No module named 'pefile'
DEBUG    volatility3.framework: Failed to import module volatility3.plugins.windows.verinfo based on file: /tmp/_MEIm9UJdo/volatility3/framework/plugins/windows/verinfo.py
INFO     volatility3.plugins.windows.verinfo: Python pefile module not found, plugin (and dependent plugins) not available
DEBUG    volatility3.framework: No module named 'pefile'
DEBUG    volatility3.framework: Failed to import module volatility3.plugins.windows.netstat based on file: /tmp/_MEIm9UJdo/volatility3/framework/plugins/windows/netstat.py
INFO     volatility3.plugins.windows.verinfo: Python pefile module not found, plugin (and dependent plugins) not available
DEBUG    volatility3.framework: No module named 'pefile'
DEBUG    volatility3.framework: Failed to import module volatility3.plugins.windows.netscan based on file: /tmp/_MEIm9UJdo/volatility3/framework/plugins/windows/netscan.py
INFO     volatility3.cli: The following plugins could not be loaded (use -vv to see why): volatility3.plugins.windows.netscan, volatility3.plugins.windows.netstat, volatility3.plugins.windows.skeleton_key_check, volatility3.plugins.windows.verinfo
usage: volatility [-h] [-c CONFIG] [--parallelism [{processes,threads,off}]] [-e EXTEND] [-p PLUGIN_DIRS] [-s SYMBOL_DIRS] [-v] [-l LOG] [-o OUTPUT_DIR] [-q] [-r RENDERER] [-f FILE]
                  [--write-config] [--save-config SAVE_CONFIG] [--clear-cache] [--cache-path CACHE_PATH] [--offline] [--single-location SINGLE_LOCATION] [--stackers [STACKERS ...]]
                  [--single-swap-locations [SINGLE_SWAP_LOCATIONS ...]]
                  plugin ...

An open-source memory forensics framework

options:
  -h, --help            Show this help message and exit, for specific plugin options use 'volatility <pluginname> --help'
  -c CONFIG, --config CONFIG
                        Load the configuration from a json file
  --parallelism [{processes,threads,off}]
                        Enables parallelism (defaults to off if no argument given)
  -e EXTEND, --extend EXTEND
                        Extend the configuration with a new (or changed) setting
  -p PLUGIN_DIRS, --plugin-dirs PLUGIN_DIRS
                        Semi-colon separated list of paths to find plugins
  -s SYMBOL_DIRS, --symbol-dirs SYMBOL_DIRS
                        Semi-colon separated list of paths to find symbols
  -v, --verbosity       Increase output verbosity
  -l LOG, --log LOG     Log output to a file as well as the console
  -o OUTPUT_DIR, --output-dir OUTPUT_DIR
                        Directory in which to output any generated files
  -q, --quiet           Remove progress feedback
  -r RENDERER, --renderer RENDERER
                        Determines how to render the output (quick, none, csv, pretty, json, jsonl)
  -f FILE, --file FILE  Shorthand for --single-location=file:// if single-location is not defined
  --write-config        Write configuration JSON file out to config.json
  --save-config SAVE_CONFIG
                        Save configuration JSON file to a file
  --clear-cache         Clears out all short-term cached items
  --cache-path CACHE_PATH
                        Change the default path (/home/pikachu/.cache/volatility3) used to store the cache
  --offline             Do not search online for additional JSON files
  --single-location SINGLE_LOCATION
                        Specifies a base location on which to stack
  --stackers [STACKERS ...]
                        List of stackers
  --single-swap-locations [SINGLE_SWAP_LOCATIONS ...]
                        Specifies a list of swap layer URIs for use with single-location

Plugins:
  For plugin specific options, run 'volatility <plugin> --help'

  plugin
    banners.Banners     Attempts to identify potential linux banners in an image
    configwriter.ConfigWriter
                        Runs the automagics and both prints and outputs configuration in the output directory.
    frameworkinfo.FrameworkInfo
                        Plugin to list the various modular components of Volatility
    isfinfo.IsfInfo     Determines information about the currently available ISF files, or a specific one
    layerwriter.LayerWriter
                        Runs the automagics and writes out the primary layer produced by the stacker.
    linux.bash.Bash     Recovers bash command history from memory.
    linux.check_afinfo.Check_afinfo
                        Verifies the operation function pointers of network protocols.
    linux.check_creds.Check_creds
                        Checks if any processes are sharing credential structures
    linux.check_idt.Check_idt
                        Checks if the IDT has been altered
    linux.check_modules.Check_modules
                        Compares module list to sysfs info, if available
    linux.check_syscall.Check_syscall
                        Check system call table for hooks.
    linux.elfs.Elfs     Lists all memory mapped ELF files for all processes.
    linux.keyboard_notifiers.Keyboard_notifiers
                        Parses the keyboard notifier call chain
    linux.kmsg.Kmsg     Kernel log buffer reader
    linux.lsmod.Lsmod   Lists loaded kernel modules.
    linux.lsof.Lsof     Lists all memory maps for all processes.
    linux.malfind.Malfind
                        Lists process memory ranges that potentially contain injected code.
    linux.mountinfo.MountInfo
                        Lists mount points on processes mount namespaces
    linux.proc.Maps     Lists all memory maps for all processes.
    linux.psaux.PsAux   Lists processes with their command line arguments
    linux.pslist.PsList
                        Lists the processes present in a particular linux memory image.
    linux.pstree.PsTree
                        Plugin for listing processes in a tree based on their parent process ID.
    linux.tty_check.tty_check
                        Checks tty devices for hooks
    mac.bash.Bash       Recovers bash command history from memory.
    mac.check_syscall.Check_syscall
                        Check system call table for hooks.
    mac.check_sysctl.Check_sysctl
                        Check sysctl handlers for hooks.
    mac.check_trap_table.Check_trap_table
                        Check mach trap table for hooks.
    mac.ifconfig.Ifconfig
                        Lists network interface information for all devices
    mac.kauth_listeners.Kauth_listeners
                        Lists kauth listeners and their status
    mac.kauth_scopes.Kauth_scopes
                        Lists kauth scopes and their status
    mac.kevents.Kevents
                        Lists event handlers registered by processes
    mac.list_files.List_Files
                        Lists all open file descriptors for all processes.
    mac.lsmod.Lsmod     Lists loaded kernel modules.
    mac.lsof.Lsof       Lists all open file descriptors for all processes.
    mac.malfind.Malfind
                        Lists process memory ranges that potentially contain injected code.
    mac.mount.Mount     A module containing a collection of plugins that produce data typically found in Mac's mount command
    mac.netstat.Netstat
                        Lists all network connections for all processes.
    mac.proc_maps.Maps  Lists process memory ranges that potentially contain injected code.
    mac.psaux.Psaux     Recovers program command line arguments.
    mac.pslist.PsList   Lists the processes present in a particular mac memory image.
    mac.pstree.PsTree   Plugin for listing processes in a tree based on their parent process ID.
    mac.socket_filters.Socket_filters
                        Enumerates kernel socket filters.
    mac.timers.Timers   Check for malicious kernel timers.
    mac.trustedbsd.Trustedbsd
                        Checks for malicious trustedbsd modules
    mac.vfsevents.VFSevents
                        Lists processes that are filtering file system events
    timeliner.Timeliner
                        Runs all relevant plugins that provide time related information and orders the results by time.
    windows.bigpools.BigPools
                        List big page pools.
    windows.cachedump.Cachedump
                        Dumps lsa secrets from memory
    windows.callbacks.Callbacks
                        Lists kernel callbacks and notification routines.
    windows.cmdline.CmdLine
                        Lists process command line arguments.
    windows.crashinfo.Crashinfo
    windows.devicetree.DeviceTree
                        Listing tree based on drivers and attached devices in a particular windows memory image.
    windows.dlllist.DllList
                        Lists the loaded modules in a particular windows memory image.
    windows.driverirp.DriverIrp
                        List IRPs for drivers in a particular windows memory image.
    windows.drivermodule.DriverModule
                        Determines if any loaded drivers were hidden by a rootkit
    windows.driverscan.DriverScan
                        Scans for drivers present in a particular windows memory image.
    windows.dumpfiles.DumpFiles
                        Dumps cached file contents from Windows memory samples.
    windows.envars.Envars
                        Display process environment variables
    windows.filescan.FileScan
                        Scans for file objects present in a particular windows memory image.
    windows.getservicesids.GetServiceSIDs
                        Lists process token sids.
    windows.getsids.GetSIDs
                        Print the SIDs owning each process
    windows.handles.Handles
                        Lists process open handles.
    windows.hashdump.Hashdump
                        Dumps user hashes from memory
    windows.info.Info   Show OS & kernel details of the memory sample being analyzed.
    windows.joblinks.JobLinks
                        Print process job link information
    windows.ldrmodules.LdrModules
    windows.lsadump.Lsadump
                        Dumps lsa secrets from memory
    windows.malfind.Malfind
                        Lists process memory ranges that potentially contain injected code.
    windows.mbrscan.MBRScan
                        Scans for and parses potential Master Boot Records (MBRs)
    windows.memmap.Memmap
                        Prints the memory map
    windows.mftscan.MFTScan
                        Scans for MFT FILE objects present in a particular windows memory image.
    windows.modscan.ModScan
                        Scans for modules present in a particular windows memory image.
    windows.modules.Modules
                        Lists the loaded kernel modules.
    windows.mutantscan.MutantScan
                        Scans for mutexes present in a particular windows memory image.
    windows.poolscanner.PoolScanner
                        A generic pool scanner plugin.
    windows.privileges.Privs
                        Lists process token privileges
    windows.pslist.PsList
                        Lists the processes present in a particular windows memory image.
    windows.psscan.PsScan
                        Scans for processes present in a particular windows memory image.
    windows.pstree.PsTree
                        Plugin for listing processes in a tree based on their parent process ID.
    windows.registry.certificates.Certificates
                        Lists the certificates in the registry's Certificate Store.
    windows.registry.hivelist.HiveList
                        Lists the registry hives present in a particular memory image.
    windows.registry.hivescan.HiveScan
                        Scans for registry hives present in a particular windows memory image.
    windows.registry.printkey.PrintKey
                        Lists the registry keys under a hive or specific key value.
    windows.registry.userassist.UserAssist
                        Print userassist registry keys and information.
    windows.sessions.Sessions
                        lists Processes with Session information extracted from Environmental Variables
    windows.ssdt.SSDT   Lists the system call table.
    windows.statistics.Statistics
    windows.strings.Strings
                        Reads output from the strings command and indicates which process(es) each string belongs to.
    windows.svcscan.SvcScan
                        Scans for windows services.
    windows.symlinkscan.SymlinkScan
                        Scans for links present in a particular windows memory image.
    windows.vadinfo.VadInfo
                        Lists process memory ranges.
    windows.vadwalk.VadWalk
                        Walk the VAD tree.
    windows.vadyarascan.VadYaraScan
                        Scans all the Virtual Address Descriptor memory maps using yara.
    windows.virtmap.VirtMap
                        Lists virtual mapped sections.
    yarascan.YaraScan   Scans kernel memory using yara rules (string or file).

The following plugins could not be loaded (use -vv to see why): volatility3.plugins.windows.netscan, volatility3.plugins.windows.netstat, volatility3.plugins.windows.skeleton_key_check,
volatility3.plugins.windows.verinfo
ikelos commented 1 year ago 
DEBUG    volatility3.framework: No module named 'pefile'
DEBUG    volatility3.framework: Failed to import module volatility3.plugins.windows.netstat based on file: /tmp/_MEIm9UJdo/volatility3/framework/plugins/windows/netstat.py

This shows that pefile was not correctly installed, and looking back at your pip line making use of the requirements, it failed on installing snappy. Please either try removing the snappy requirement, or trying out #913, then make sure all the requirements are successfully installed and try again.

github-actions[bot] commented 10 months ago

This issue is stale because it has been open for 200 days with no activity.

github-actions[bot] commented 8 months ago

This issue was closed because it has been inactive for 60 days since being marked as stale.