qiling fails to install due to an error installing pyelftools - No such file or directory: '/usr/local/lib/python3.8/dist-packages/pyelftools-0.29.dist-info/METADATA'"

[seanthegeek]

  pip_|-remnux-python3-packages-qiling_|-qiling_|-installed:
    __id__: remnux-python3-packages-qiling
    __run_num__: 387
    __sls__: remnux.python3-packages.qiling
    changes: {}
    comment: "Failed to install packages: qiling. Error: Requirement already satisfied:\
      \ qiling in /usr/local/lib/python3.8/dist-packages (1.4.5)\nCollecting qiling\n\
      \  Obtaining dependency information for qiling from https://files.pythonhosted.org/packages/7b/ac/3b72bd96612a2d3af84426533c915de847e481cb6819be824eb967ca6ca9/qiling-1.4.6-py3-none-any.whl.metadata\n\
      \  Downloading qiling-1.4.6-py3-none-any.whl.metadata (12 kB)\nRequirement already\
      \ satisfied: capstone>=4.0.1 in /usr/local/lib/python3.8/dist-packages (from\
      \ qiling) (4.0.2)\nCollecting unicorn>=2.0.1 (from qiling)\n  Downloading unicorn-2.0.1.post1-py2.py3-none-manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl\
      \ (16.1 MB)\n     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.1/16.1 MB 14.5\
      \ MB/s eta 0:00:00\nRequirement already satisfied: pefile>=2022.5.30 in /usr/local/lib/python3.8/dist-packages\
      \ (from qiling) (2023.2.7)\nRequirement already satisfied: python-registry>=1.3.1\
      \ in /usr/local/lib/python3.8/dist-packages (from qiling) (1.3.1)\nRequirement\
      \ already satisfied: keystone-engine>=0.9.2 in /usr/local/lib/python3.8/dist-packages\
      \ (from qiling) (0.9.2)\nRequirement already satisfied: pyelftools>=0.28 in\
      \ /usr/local/lib/python3.8/dist-packages (from qiling) (0.29)\nRequirement already\
      \ satisfied: gevent>=20.9.0 in /usr/local/lib/python3.8/dist-packages (from\
      \ qiling) (21.12.0)\nRequirement already satisfied: multiprocess>=0.70.12.2\
      \ in /usr/local/lib/python3.8/dist-packages (from qiling) (0.70.12.2)\nRequirement\
      \ already satisfied: pyyaml>=6.0 in /usr/local/lib/python3.8/dist-packages (from\
      \ qiling) (6.0.1)\nRequirement already satisfied: python-fx in /usr/local/lib/python3.8/dist-packages\
      \ (from qiling) (0.2.0)\nRequirement already satisfied: questionary in /usr/local/lib/python3.8/dist-packages\
      \ (from qiling) (1.10.0)\nRequirement already satisfied: termcolor in /usr/local/lib/python3.8/dist-packages\
      \ (from qiling) (1.1.0)\nRequirement already satisfied: zope.event in /usr/local/lib/python3.8/dist-packages\
      \ (from gevent>=20.9.0->qiling) (4.5.0)\nRequirement already satisfied: zope.interface\
      \ in /usr/local/lib/python3.8/dist-packages (from gevent>=20.9.0->qiling) (6.0)\n\
      Requirement already satisfied: setuptools in /usr/local/lib/python3.8/dist-packages\
      \ (from gevent>=20.9.0->qiling) (60.9.3)\nRequirement already satisfied: greenlet<2.0,>=1.1.0\
      \ in /usr/local/lib/python3.8/dist-packages (from gevent>=20.9.0->qiling) (1.1.2)\n\
      Requirement already satisfied: dill>=0.3.4 in /usr/local/lib/python3.8/dist-packages\
      \ (from multiprocess>=0.70.12.2->qiling) (0.3.4) ERROR: Could not install packages\
      \ due to an OSError: [Errno 2] No such file or directory: '/usr/local/lib/python3.8/dist-packages/pyelftools-0.29.dist-info/METADATA'"
    duration: 2299.226
    name: qiling
    result: false
    start_time: '13:04:04.884403'

[seanthegeek]

Weird. Even after upgrading setuptools pip and wheel as I described here, it still won't install.

freeze.txt saltstack.log

[digitalsleuth]

Everything actually worked correctly, the error was a temporary domain resolution failure with github.com:

Line 27113: # [ERROR   ] Command '/usr/bin/python3' failed with return code: 1
Line 27114: # [ERROR   ] stdout: Collecting git+https://github.com/obsidianforensics/unfurl.git
Line 27116: # [ERROR   ] stderr:   Running command git clone --filter=blob:none --quiet https://github.com/obsidianforensics/unfurl.git /tmp/pip-req-build-af9zi5um
 Cloning https://github.com/obsidianforensics/unfurl.git to /tmp/pip-req-build-af9zi5um
# [ERROR   ] stderr:   Running command git clone --filter=blob:none --quiet https://github.com/obsidianforensics/unfurl.git /tmp/pip-req-build-af9zi5um
  fatal: unable to access 'https://github.com/obsidianforensics/unfurl.git/': Could not resolve host: github.com
  error: subprocess-exited-with-error

  × git clone --filter=blob:none --quiet https://github.com/obsidianforensics/unfurl.git /tmp/pip-req-build-af9zi5um did not run successfully.
  │ exit code: 128
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× git clone --filter=blob:none --quiet https://github.com/obsidianforensics/unfurl.git /tmp/pip-req-build-af9zi5um did not run successfully.
│ exit code: 128
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
Line 27132: # [ERROR   ] retcode: 1
Line 27133: # [ERROR   ] Failed to install packages: git+https://github.com/obsidianforensics/unfurl.git. Error: Collecting git+https://github.com/obsidianforensics/unfurl.git

[seanthegeek]

D'oh! I don't know how I missed that for unfurl. After I tried the upgrade again it was fine.

That said, I forgot to mention what I had to do to get the Python packages to install correctly first.

Even when using --force-reinstall, pip would still install an outdated version of setuptools

sudo pip install --force-reinstall --break-system-packages -U pip setuptools wheel

Collecting pip
  Obtaining dependency information for pip from https://files.pythonhosted.org/packages/50/c2/e06851e8cc28dcad7c155f4753da8833ac06a5c704c109313b8d5a62968a/pip-23.2.1-py3-none-any.whl.metadata
  Downloading pip-23.2.1-py3-none-any.whl.metadata (4.2 kB)
Collecting setuptools
  Obtaining dependency information for setuptools from https://files.pythonhosted.org/packages/bb/26/7945080113158354380a12ce26873dd6c1ebd88d47f5bc24e2c5bb38c16a/setuptools-68.2.2-py3-none-any.whl.metadata
  Downloading setuptools-68.2.2-py3-none-any.whl.metadata (6.3 kB)
Collecting wheel
  Obtaining dependency information for wheel from https://files.pythonhosted.org/packages/b8/8b/31273bf66016be6ad22bb7345c37ff350276cfd46e389a0c2ac5da9d9073/wheel-0.41.2-py3-none-any.whl.metadata
  Downloading wheel-0.41.2-py3-none-any.whl.metadata (2.2 kB)
Downloading pip-23.2.1-py3-none-any.whl (2.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 8.6 MB/s eta 0:00:00
Downloading setuptools-68.2.2-py3-none-any.whl (807 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 807.9/807.9 kB 16.6 MB/s eta 0:00:00
Downloading wheel-0.41.2-py3-none-any.whl (64 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.8/64.8 kB 15.7 MB/s eta 0:00:00
Installing collected packages: wheel, setuptools, pip
  Attempting uninstall: wheel
    Found existing installation: wheel 0.36.2
    Uninstalling wheel-0.36.2:
      Successfully uninstalled wheel-0.36.2
  Attempting uninstall: setuptools
    Found existing installation: setuptools 60.7.1
    Uninstalling setuptools-60.7.1:
      Successfully uninstalled setuptools-60.7.1
  Attempting uninstall: pip
    Found existing installation: pip 23.2
    Uninstalling pip-23.2:
      Successfully uninstalled pip-23.2
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
launchpadlib 1.10.13 requires testresources, which is not installed.
speakeasy-emulator 1.5.11 requires unicorn==1.0.2, but you have unicorn 2.0.1.post1 which is incompatible.
thug 5.0.1 requires charset-normalizer==3.1.0, but you have charset-normalizer 2.1.1 which is incompatible.
thug 5.0.1 requires cssutils==2.6.0, but you have cssutils 2.7.1 which is incompatible.
thug 5.0.1 requires requests-futures==1.0.0, but you have requests-futures 1.0.1 which is incompatible.
Successfully installed pip-23.2.1 setuptools-60.8.0 wheel-0.41.2

It turns out that I somehow had multiple versions of setuptools installed

ls /usr/local/lib/python3.8/dist-packages/setuptools
setuptools/                      setuptools-68.0.0.dist-info/
setuptools-61.3.0.dist-info/     setuptools-68.2.2.dist-info/
setuptools-65.7.0.dist-info/     setuptools_rust/
setuptools-67.8.0.dist-info/     setuptools_rust-1.7.0.dist-info/

Once I manually removed those, I was able to upgrade setuptools

sudo rm -rf /usr/local/lib/python3.8/dist-packages/setuptools-*

sudo pip install --break-system-packages -U pip setuptools wheel

sudo pip install --force-reinstall --break-system-packages -U pip setuptools wheel
Collecting pip
  Obtaining dependency information for pip from https://files.pythonhosted.org/packages/50/c2/e06851e8cc28dcad7c155f4753da8833ac06a5c704c109313b8d5a62968a/pip-23.2.1-py3-none-any.whl.metadata
  Using cached pip-23.2.1-py3-none-any.whl.metadata (4.2 kB)
Collecting setuptools
  Obtaining dependency information for setuptools from https://files.pythonhosted.org/packages/bb/26/7945080113158354380a12ce26873dd6c1ebd88d47f5bc24e2c5bb38c16a/setuptools-68.2.2-py3-none-any.whl.metadata
  Using cached setuptools-68.2.2-py3-none-any.whl.metadata (6.3 kB)
Collecting wheel
  Obtaining dependency information for wheel from https://files.pythonhosted.org/packages/b8/8b/31273bf66016be6ad22bb7345c37ff350276cfd46e389a0c2ac5da9d9073/wheel-0.41.2-py3-none-any.whl.metadata
  Using cached wheel-0.41.2-py3-none-any.whl.metadata (2.2 kB)
Using cached pip-23.2.1-py3-none-any.whl (2.1 MB)
Using cached setuptools-68.2.2-py3-none-any.whl (807 kB)
Using cached wheel-0.41.2-py3-none-any.whl (64 kB)
Installing collected packages: wheel, setuptools, pip
  Attempting uninstall: wheel
    Found existing installation: wheel 0.41.2
    Uninstalling wheel-0.41.2:
      Successfully uninstalled wheel-0.41.2
  Attempting uninstall: setuptools
    Found existing installation: setuptools 52.0.0
    Uninstalling setuptools-52.0.0:
      Successfully uninstalled setuptools-52.0.0
  Attempting uninstall: pip
    Found existing installation: pip 23.2.1
    Uninstalling pip-23.2.1:
      Successfully uninstalled pip-23.2.1

It was a similar situation with pyelftools

ls /usr/local/lib/python3.8/dist-packages/pyelftools*
pyelftools-0.29.dist-info/ pyelftools-0.30.dist-info/
sudo rm -rf /usr/local/lib/python3.8/dist-packages/pyelftools-*
sudo pip install --force-reinstall --break-system-packages -U pyelftools

I removed unneeded system packages.

sudo apt autoremove -y

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  python3-contextvars python3-croniter python3-gnupg
  python3-immutables python3-jinja2 python3-markupsafe
  python3-msgpack python3-pycryptodome python3-tz python3-zmq
0 upgraded, 0 newly installed, 10 to remove and 0 not upgraded.
After this operation, 22.8 MB disk space will be freed.
(Reading database ... 406430 files and directories currently instal
led.)
Removing python3-contextvars (2.4-1) ...
Removing python3-croniter (0.3.29-2ubuntu1) ...
Removing python3-gnupg (0.4.5-2) ...
Removing python3-immutables (0.14-1) ...
Removing python3-jinja2 (2.11.3-1ppa1~focal) ...
Removing python3-markupsafe (1.1.1-1ppa1~focal) ...
Removing python3-msgpack (0.6.2-1) ...
Removing python3-pycryptodome (3.6.1-2build4) ...
Removing python3-tz (2022.1-1ppa1~focal) ...
Removing python3-zmq (23.2.0-1ppa1~focal) ..

Then I was able to sucesssfully run remnux uograde

[digitalsleuth]

@seanthegeek That's good to hear. I've just discovered that setuptools 68+ has introduced a new set of problems, so the version will be downgrading in the future to resolve this (and the inability to uninstall). Cheers!