search

REMnux / remnux-cli

This repository contains the source code for the REMnux installer, which is the command-line tool for installing and upgrading the REMnux distro.
https://REMnux.org
MIT License
38 stars 11 forks source link

Dependency Discrepency? #148

Closed geeksailor closed 1 year ago

geeksailor commented 1 year ago

Hello, I attempted to do an remnux upgrade today and ran into issues with pyelftool. I grep'ed the saltstack.log and came up with:

cat saltstack.log | grep -i -B 8 'result: false' file|-remnux-addon-version-file|-/etc/remnux-version_|-managed: id: remnux-addon-version-file __run_num: 779 sls__: remnux.addon changes: {} comment: 'One or more requisite failed: remnux.python3-packages.remnux-python3-packages' duration: 0.008 name: /etc/remnux-version result: false

\ packages: pyelftools\n Attempting uninstall: pyelftools\n Found existing\ \ installation: pyelftools 0.29 WARNING: Error parsing requirements for pyelftools:\ \ [Errno 2] No such file or directory: '/usr/local/lib/python3.8/dist-packages/pyelftools-0.29.dist-info/METADATA'\n\ \ WARNING: No metadata found in /usr/local/lib/python3.8/dist-packages\n\ ERROR: Cannot uninstall pyelftools 0.29, RECORD file not found. You might be\ \ able to recover from this via: 'pip install --force-reinstall --no-deps pyelftools==0.29'." duration: 3912.997 name: pyelftools result: false

test|-remnux-dedicated|-remnux-dedicated_|-nop: id: remnux-dedicated __run_num: 836 sls__: remnux.dedicated changes: {} comment: 'One or more requisite failed: remnux.addon.remnux-addon-version-file' duration: 0.009 name: remnux-dedicated result: false

test|-remnux-python3-packages|-remnux-python3-packages_|-nop: id: remnux-python3-packages __run_num: 412 sls__: remnux.python3-packages changes: {} comment: 'One or more requisite failed: remnux.python3-packages.pyelftools.remnux-python3-packages-pyelftools' duration: 0.008 name: remnux-python3-packages result: false

I ran a pip list to see what I had installed pip list | grep pyelf* and got pip list | grep pyel* pyelftools 0.30 Turns out I had fragments of a pip install of 0.29.0 and a complete install of 0.30.0. so I went to the /usr/local/lib/python3.8/dist-packages and looked in the 0.29.0 folder and all that was in there was the REQUESTED file. So I nuked the 0.29.0 folder since it does not seem to be being utilized. I ran a pip install --force-reinstall pyelftool to make sure it was good and came back with the error

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. qiling 1.4.6 requires unicorn>=2.0.1, but you have unicorn 1.0.2 which is incompatible.

Ok fine, unicorn needs fixing. Ran pip list | grep unicorn to make sure what I have and:

pip list | grep uni* unicorn 1.0.2

Got to be kidding me. Ok, lets upgrade that then. Ran a pip install --force-reinstall unicorn. Then came back with

Defaulting to user installation because normal site-packages is not writeable Collecting unicorn Downloading unicorn-2.0.1.post1-py2.py3-none-manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x8664.whl (16.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.1/16.1 MB 453.5 kB/s eta 0:00:00 Installing collected packages: unicorn ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. speakeasy-emulator 1.5.11 requires unicorn==1.0.2, but you have unicorn 2.0.1.post1 which is incompatible._ Successfully installed unicorn-2.0.1.post1

ran the command speakeasy by itself and came up with the menu like nothing was wrong so far. I haven't tried to run it fully yet to test if it has issues (trying to get my hands on some malware to test it with). But it looks like pyelftool requires 2.0.1 but speakeasy requires 1.0.2? Should I just install both packages to satisfy both programs or will it just confuse the system? Has anyone else ran into this issue? Thanks!

digitalsleuth commented 1 year ago

Hi @geeksailor , looks like there are a few intricacies to deal with here, including the [Errno 2] No such file or directory: '/usr/local/lib/python3.8/dist-packages/pyelftools-0.29.dist-info/METADATA error you noticed in your log. This actually might be related to something other than pyelftool, that we've been investigating.

Would you be able to tell me how you first came by your REMnux installation (built from scratch / downloaded OVA / previous REMnux install you upgraded etc), so we can find out where this issue started?

digitalsleuth commented 1 year ago

Hi @geeksailor , can you provide me the output of the following command? sudo python3 -m pip list

geeksailor commented 1 year ago

Hi @digitalsleuth! This was an install on an existing system (Ubuntu Mate 20.04.06). I believe I did the install running the remnux install --mode=addon command but I could be wrong and I might have used Cast to do the install. I've been testing different Ubuntu flavors (in a VM) and doing installs from scratch on them to see what I like in conjuction with REMnux. I've ran into install issues a few times using just the remnux install command and had to use Cast get the install to complete sucessfully. But that's a story for another time. I did run another remnux update after and it works find after I cleared out the lower version of the package that had no content so it seems that the folder with nothing in it confuses the program? Here is the pip list you requested

sudo pip3 list Package Version

aiohttp 3.8.5 aiosignal 1.3.1 aiosmtpd 1.4.4.post2 aiosmtplib 2.0.2 androguard 3.3.5 aniso8601 9.0.1 antlr4-python3-runtime 4.8 anytree 2.9.0 appdirs 1.4.4 apturl 0.5.2 asciimatics 1.14.0 asn1crypto 1.5.1 async-timeout 4.0.3 atpublic 4.0 attrs 23.1.0 backcall 0.1.0 backports.zoneinfo 0.2.1 base58 2.1.1 bcrypt 4.0.1 beautifulsoup4 4.12.2 binwalk 2.2.0 bitarray 2.8.1 blinker 1.6.2 Brlapi 0.7.0 Brotli 1.0.7 cajarename 19.7.15 capstone 5.0.1 cattrs 23.1.2 certifi 2023.7.22 cffi 1.15.1 chardet 3.0.4 charset-normalizer 3.2.0 chepy 5.1.0 chrome-gnome-shell 0.0.0 click 8.1.3 colorama 0.4.6 colorclass 2.2.2 command-not-found 0.3 compressed-rtf 1.0.6 configobj 5.0.6 configparser 6.0.0 construct 2.9.45 coverage 7.3.0 crccheck 1.3.0 cryptography 39.0.2 cssutils 2.7.1 cupshelpers 1.0 cxxfilt 0.2.2 cycler 0.10.0 dacite 1.6.0 dbus-python 1.2.16 debloat 1.5.3.2 decorator 5.1.1 defer 1.0.6 defusedxml 0.7.1 dfir-unfurl 20221100 dhash 1.4 dill 0.3.7 distlib 0.3.0 distro 1.8.0 distro-info 0.23+ubuntu1.1 dnfile 0.14.1 dnslib 0.9.23 dnspython 2.0.0 docker 6.1.3 docker-compose 1.29.2 dockerpty 0.4.1 docopt 0.6.2 docstring-parser 0.15 dotnetfile 0.2.4 dpkt 1.9.8 droidlysis 3.4.5 easygui 0.98.1 ebcdic 1.1.1 EditorConfig 0.12.3 emoji 2.0.0 entrypoints 0.3 enum-compat 0.0.3 exceptiongroup 1.1.3 execnet 2.0.2 exrex 0.11.0 extract-msg 0.45.0 fakemail 0.1.0 FakeNet-NG 3.0 fasttext 0.9.2 filelock 3.0.12 filetype 1.2.0 fire 0.4.0 first 2.0.2 Flask 2.3.3 Flask-Cors 4.0.0 flask-restx 1.1.0 flatbencode 0.2.1 folder-color-caja 0.0.86 folder-color-common 0.0.86 fonttools 4.5.0 frida 16.1.4 frida-tools 12.2.1 frozenlist 1.4.0 fs 2.4.11 future 0.18.2 gevent 23.9.0.post1 gpg 1.13.1 greenlet 2.0.2 hachoir 3.2.0 hexdump 3.3 html5lib 1.1 httplib2 0.14.0 idna 3.4 IMAPClient 2.3.1 importlib-metadata 4.13.0 importlib-resources 6.0.1 iniconfig 2.0.0 ipwhois 1.2.0 ipython 7.13.0 ipython_genutils 0.2.0 isodate 0.6.1 itsdangerous 2.1.2 javaobj-py3 0.4.3 jedi 0.15.2 Jinja2 3.0.3 joblib 0.16.0 jsbeautifier 1.14.9 jsonpath-ng 1.5.3 jsonpickle 3.0.2 jsonschema 3.2.0 jsonschema-extractor 2.0.2 jsonschema-specifications 2023.7.1 keyring 18.0.1 keystone-engine 0.9.2 kiwisolver 1.0.1 language-selector 0.1 lark 1.1.5 lark-parser 0.12.0 launchpadlib 1.10.13 lazr.restfulclient 0.14.2 lazr.uri 1.0.3 lazy-import 0.2.2 leechcorepyc 2.16.1 libcsce 0.1.0 lightgbm 3.3.5 loguru 0.6.0 louis 3.12.0 lxml 4.9.3 lznt1 0.2 M2Crypto 0.39.0 macaroonbakery 1.3.1 maclookup 1.0.3 Magnus 1.0.3 mail-parser 3.15.0 malwareconfig 1.0.4 markdown-it-py 3.0.0 MarkupSafe 2.1.3 mate-hud 19.10.0 mate-menu 20.4.1 mate-tweak 20.4.0 matplotlib 3.1.2 mdurl 0.1.2 minidump 0.0.21 more-itertools 4.2.0 mpmath 1.1.0 msgpack 1.0.5 msoffcrypto-tool 5.1.1 multidict 6.0.4 multiprocess 0.70.15 mwcp 3.13.0 name-that-hash 1.11.0 NetfilterQueue 1.1.0 netifaces 0.10.4 networkx 3.1 numpy 1.24.4 oauthlib 3.1.0 olefile 0.46 oletools 0.60.1 onboard 1.4.1 overrides 6.2.0 packaging 23.1 pandas 2.0.3 paramiko 3.3.1 parso 0.5.2 pbkdf2 1.3 pcode2code 0.1.1 pcodedmp 1.2.6 pdfminer 20191125 pe-tree 1.0.30 peepdf 1.0.5 pefile 2023.2.7 peframe-ds 6.1.0 pexpect 4.6.0 pickleshare 0.7.5 Pillow 9.2.0 pip 23.2.1 pipx 0.12.3.1 pkgutil_resolve_name 1.3.10 platformdirs 3.10.0 pluggy 1.3.0 ply 3.11 promise 2.3 prompt-toolkit 2.0.10 protobuf 3.20.3 psutil 5.9.1 publicsuffix2 2.20191221 pulsemixer 1.5.0 pyasn1 0.4.8 pyasn1-modules 0.2.8 pybind11 2.11.1 pycairo 1.16.2 pycdlib 1.14.0 pycipher 0.5.2 pycountry 22.3.5 pycparser 2.21 pycrypto 2.6.1 pycryptodome 3.9.7 pycryptodomex 3.7.3 pycups 1.9.73 pydash 7.0.6 pydivert 2.1.0 pydot 1.4.2 pyelftools 0.30 pyfiglet 0.8.post1 pyftpdlib 1.5.7 Pygments 2.16.1 PyGObject 3.36.0 PyJWT 2.0.1 pylibemu 0.8 pylzma 0.5.0 pymacaroons 0.13.0 pymispwarninglists 1.5 pymongo 4.4.1 PyNaCl 1.5.0 PyOpenGL 3.1.0 pyOpenSSL 23.2.0 pyparsing 2.4.7 PyPDF2 3.0.1 pyperclip 1.8.2 PyQt5 5.15.9 PyQt5-Qt5 5.15.2 PyQt5-sip 12.12.2 pyqtgraph 0.11.0rc0 PyQtWebEngine 5.15.6 PyQtWebEngine-Qt5 5.15.2 pyRFC3339 1.1 pyrsistent 0.19.3 PySocks 1.7.1 pytesseract 0.3.10 pytest 7.4.1 pytest-cov 4.1.0 pytest-datadir 1.4.1 pytest-mock 3.11.1 pytest-xdist 3.3.1 python-apt 2.0.1+ubuntu0.20.4.1 python-dateutil 2.8.2 python-debian 0.1.36+ubuntu1.1 python-dotenv 0.21.1 python-editor 1.0.4 python-fx 0.2.0 python-magic 0.4.12 python-registry 1.3.1 python-xapp 1.8.1 python-xlib 0.23 pythonaes 1.0 pytz 2023.3.post1 pyxdg 0.26 pyxlsb2 0.0.9 PyYAML 6.0.1 pyzipper 0.3.6 qiling 1.4.6 questionary 2.0.0 r2pipe 1.8.0 rarfile 4.0 red-black-tree-mod 1.20 referencing 0.30.2 regex 2023.8.8 reportlab 3.5.34 requests 2.31.0 requests-futures 1.0.1 requests-unixsocket 0.2.0 rich 13.5.2 roman 4.1 rpds-py 0.10.2 RTFDE 0.1.0 ruamel.yaml 0.17.32 ruamel.yaml.clib 0.2.7 scandir 1.10.0 scikit-learn 0.23.2 scipy 1.3.3 SecretStorage 2.3.1 semantic-version 2.10.0 setproctitle 1.1.10 setuptools 65.7.0 setuptools-rust 1.7.0 simplejson 3.19.1 sip 4.19.21 six 1.16.0 soupsieve 2.5 speakeasy-emulator 1.5.11 SQLAlchemy 2.0.20 ssdeep 3.4 ssh-import-id 5.10 stix2 3.0.1 stix2-patterns 2.0.0 stpyv8 10.1.124.12 stringsifter 2.20201202 sympy 1.5.1 systemd-python 234 tabulate 0.9.0 termcolor 2.3.0 texttable 1.6.7 threadpoolctl 3.2.0 thug 5.4 time-decode 6.1.0 tkinterdnd2 0.3.0 tomli 2.0.1 torbrowser-launcher 0.3.2 torf 4.2.1 traitlets 4.3.3 typing_extensions 4.7.1 tzdata 2023.3 tzlocal 5.0.1 ubuntu-advantage-tools 8001 ubuntu-drivers-common 0.0.0 ufw 0.36 ulid-py 1.1.0 unicode 2.9 unicodecsv 0.14.1 unicorn 1.0.2 untangle 1.2.1 urllib3 2.0.4 urwid 2.1.2 vboxapi 1.0 virtualenv 20.0.17 vivisect 1.1.1 volatility3 2.5.1 wadllib 1.3.3 wcwidth 0.2.5 webencodings 0.5.1 websocket-client 0.59.0 Werkzeug 2.3.7 wheel 0.36.2 xkit 0.0.0 XLMMacroDeobfuscator 0.2.7 xlrd2 1.3.4 xortool 1.0.2 xxxswf 2.0.1 yamale 4.0.4 yara-python 4.3.1 yarl 1.9.2 zipp 3.16.2 zope.event 5.0 zope.interface 6.0

digitalsleuth commented 1 year ago

@geeksailor , it looks like this issue stems from another issue with setuptools and wheel. I'm currently working on a solution for it and hope to have something concrete within the next 24 hours.

geeksailor commented 1 year ago

Sounds good! Thanks @digitalsleuth

geeksailor commented 1 year ago

Morning @digitalsleuth! Just waned to let you know that I ran a remnux update this morning and it tried to do an upgrade from 35.1 -> 36.1 but it failed due to pyelftools. It was looking for v0.29.0 to uninstall but since it doesn't exist it caused it to fail.

cat saltstack.log | grep -i -B 8 "result: false" file|-remnux-addon-version-file|-/etc/remnux-version_|-managed: id: remnux-addon-version-file __run_num: 780 sls__: remnux.addon changes: {} comment: 'One or more requisite failed: remnux.python3-packages.remnux-python3-packages' duration: 0.01 name: /etc/remnux-version result: false

\ packages: pyelftools\n Attempting uninstall: pyelftools\n Found existing\ \ installation: pyelftools 0.29 WARNING: Error parsing requirements for pyelftools:\ \ [Errno 2] No such file or directory: '/usr/local/lib/python3.8/dist-packages/pyelftools-0.29.dist-info/METADATA'\n\ \ WARNING: No metadata found in /usr/local/lib/python3.8/dist-packages\n\ ERROR: Cannot uninstall pyelftools 0.29, RECORD file not found. You might be\ \ able to recover from this via: 'pip install --force-reinstall --no-deps pyelftools==0.29'." duration: 4991.226 name: pyelftools result: false

test|-remnux-dedicated|-remnux-dedicated_|-nop: id: remnux-dedicated __run_num: 837 sls__: remnux.dedicated changes: {} comment: 'One or more requisite failed: remnux.addon.remnux-addon-version-file' duration: 0.008 name: remnux-dedicated result: false

test|-remnux-python3-packages|-remnux-python3-packages_|-nop: id: remnux-python3-packages __run_num: 413 sls__: remnux.python3-packages changes: {} comment: 'One or more requisite failed: remnux.python3-packages.pyelftools.remnux-python3-packages-pyelftools' duration: 0.007 name: remnux-python3-packages result: false

However, I went into traversed into the folder where the salts are stored /var/cache/remnux/cli/v2023.36.1/saltstates-2023.36.1/remnux/python3-packages/ (also checked 35.1) and looked at the pyelftool salt and I don't see anything specifying a required version of it in the salt. Am I missing something on what may be making it look for that specific version of that module? I'll try doing a force reinstall in the meantime rolling it back to 0.29.0 to see if it fixes it but not sure if something is forcing it to specifically use 0.29.0?

digitalsleuth commented 1 year ago

Hi @geeksailor , the issue isn't pyelftools, but it's the installer dependencies (pip, setuptools, wheel) which seem to be causing the problem. Could you try the following to confirm, in this order: sudo python3 -m pip install --upgrade pyelftools This should fail. Next run: sudo python3 -m pip install --upgrade pip sudo python3 -m pip install --upgrade pyelftools The pyelftools command may fail. If so, continue with the following: sudo python3 -m pip install --upgrade setuptools sudo python3 -m pip install --upgrade pyelftools If this fails, then the issue may actually be wheel: sudo python3 -m pip install --upgrade wheel sudo python3 -m pip install --upgrade pyelftools

If you wouldn't mind running all three to confirm where the error gets fixed I'd appreciate it. I've got a PR ready to be submitted which SHOULD fix this, but I just needed another real-world example, because I've been finding it difficult to reliably duplicate the error.

geeksailor commented 1 year ago

Running now but it gives a the suggestion to run pip install --force-reinstall --no-deps pyelftools==0.29 which would upgrade Python2's pip if I'm not mistaken. I don't know off the top of my head but does Python3 pip read both 2 and 3? Could it be getting confused on which pip to utilize?

geeksailor commented 1 year ago

@digitalsleuth I got the following after the first update to pyelftools:

sudo pip3 install --upgrade pyelftools Requirement already satisfied: pyelftools in /usr/local/lib/python3.8/dist-packages (0.30)

It looks like the most current version is already installed for pip3

digitalsleuth commented 1 year ago

Hi @geeksailor , the reason it's giving that suggestion is that the "pip install" is implying the currently executed version of pip, which we specified as the python3 version. If you did run pip alone, without the "python3 -m", it may revert to the python2 version; but pip isn't smart enough to know about other py versions of itself.

Can you run the remainder of the commands and let me know the output?

geeksailor commented 1 year ago

python3 -m pip list | grep pyelfpyelftools

pyelftools 0.30

python -m pip list | grep pyelf

pyelftools 0.29

pip -V

pip 23.2.1 from /usr/local/lib/python3.8/dist-packages/pip (python 3.8)

pip3 -V

pip 23.2.1 from /usr/local/lib/python3.8/dist-packages/pip (python 3.8)

pip2 -V

pip 20.3.4 from /usr/local/lib/python2.7/dist-packages/pip (python 2.7)

I think I downgraded the pip2 version but I should have downgraded the pip3 version

geeksailor commented 1 year ago

sudo python3 -m pip install --upgrade pip

Requirement already satisfied: pip in /usr/local/lib/python3.8/dist-packages (23.2.1)

sudo python3 -m pip install --upgrade pyelftools

Requirement already satisfied: pyelftools in /usr/local/lib/python3.8/dist-packages (0.30)

digitalsleuth commented 1 year ago

@geeksailor , the error has to do with python3 not python2. I'll submit a new pull request this afternoon which should fix the issue in question, and will upgrade wheel, setuptools, and pip to avoid the error you're getting.

geeksailor commented 1 year ago

@digitalsleuth Since your released a new update fixing the pyelftool issue I'm gonna close this one out as well.