`vstack` unit fails to execute on binref installations with Python 3.12 due to failed Unicorn import

cxiao commented 10 months ago

Description

When running the vstack unit on an installation of Binary Refinery in a Python virtual environment running Python 3.12, I kept seeing the following error:

failure in vstack: dependency unicorn is missing; run pip install intervaltree capstone unicorn

This was despite verifying inside the virtual environment that the listed dependencies were present.

To Reproduce

Install Binary Refinery with the binary-refinery[all] spec in an environment which uses Python 3.12, then execute the vstack unit.

Re-installing refinery inside a virtual environment with Python 3.11.6 fixes this issue.

Cause

As far as I can tell, this is the reason for this issue:

Python 3.12 removes the distutils module from the standard library (which was deprecated starting in 3.10: https://docs.python.org/3.12/whatsnew/3.10.html#distutils-deprecated)
The Unicorn bindings for Python still imports distutils: https://github.com/unicorn-engine/unicorn/blob/d4b92485b1a228fb003e1218e42f6c778c655809/bindings/python/unicorn/unicorn.py#L5
This results in a failure to import unicorn when executing the vstack unit, which results in a RefineryImportMissing exception, which is reported by refinery as a missing dependency.

Environment

Operating System: Fedora Linux 39 (which uses Python 3.12 by default)
Python Version: 3.12.0
Refinery Version: 0.6.24

Additional Context

Verbose error and traceback:

$ vstack -v -v
(13:31:14) failure in vstack: dependency unicorn is missing; run pip install unicorn capstone intervaltree
Traceback (most recent call last):
  File "/home/cxiao/.local/pipx/venvs/binary-refinery-br/lib64/python3.12/site-packages/refinery/units/__init__.py", line 1251, in __get__
    self.module = module = self.fget()
                           ^^^^^^^^^^^
  File "/home/cxiao/.local/pipx/venvs/binary-refinery-br/lib64/python3.12/site-packages/refinery/units/formats/exe/vstack.py", line 74, in _unicorn
    import unicorn
  File "/home/cxiao/.local/pipx/venvs/binary-refinery-br/lib64/python3.12/site-packages/unicorn/__init__.py", line 4, in <module>
    from .unicorn import Uc, uc_version, uc_arch_supported, version_bind, debug, UcError, __version__
  File "/home/cxiao/.local/pipx/venvs/binary-refinery-br/lib64/python3.12/site-packages/unicorn/unicorn.py", line 5, in <module>
    import distutils.sysconfig
ModuleNotFoundError: No module named 'distutils'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cxiao/.local/pipx/venvs/binary-refinery-br/lib64/python3.12/site-packages/refinery/units/__init__.py", line 1391, in normalized_action
    yield from (x for x in result if x is not None)
  File "/home/cxiao/.local/pipx/venvs/binary-refinery-br/lib64/python3.12/site-packages/refinery/units/__init__.py", line 1391, in <genexpr>
    yield from (x for x in result if x is not None)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cxiao/.local/pipx/venvs/binary-refinery-br/lib64/python3.12/site-packages/refinery/units/__init__.py", line 806, in <genexpr>
    return (Chunk.Wrap(r) for r in result)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cxiao/.local/pipx/venvs/binary-refinery-br/lib64/python3.12/site-packages/refinery/units/formats/exe/vstack.py", line 146, in process
    uc = self._unicorn
         ^^^^^^^^^^^^^
  File "/home/cxiao/.local/pipx/venvs/binary-refinery-br/lib64/python3.12/site-packages/refinery/units/__init__.py", line 1257, in __get__
    raise RefineryImportMissing(self.dependency, *args) from E
refinery.units.RefineryImportMissing

huettenhain commented 10 months ago

Ok so ... "bad" news: I cannot reproduce this in the CICD tests. And even though the coverage for vstack isn't great, I am covering that import. I am wondering if GitHub actions automagically installs distutils despite its deprecation? I will have to research more.

huettenhain commented 10 months ago

Oddly enough, on a vanilla Python 3.12 install in Windows, importing distutils just works, there is no hint of deprecation:

Python 3.12.0 (tags/v3.12.0:0fb18b0, Oct  2 2023, 13:03:39) [MSC v.1935 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import distutils
>>>

I then installed Python 3.12 on Ubuntu, and this one is even more interesting. It does not come with distutils:

Python 3.12.0 (main, Oct 21 2023, 17:44:38) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import distutils
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'distutils'
>>>

but after installing binary refinery via pip:

$ pip install binary-refinery[default]
...
$ emit AAAAAH | vstack -vv
(12:43:10) comment in vstack: mapping 65.536 kB of stack at 0x10000
(12:43:10) verbose in vstack: emulating [wait=00] 0x00000000: inc ecx
(12:43:10) verbose in vstack: emulating [wait=01] 0x00000001: inc ecx
(12:43:10) verbose in vstack: emulating [wait=02] 0x00000002: inc ecx
(12:43:10) verbose in vstack: emulating [wait=03] 0x00000003: inc ecx
(12:43:10) verbose in vstack: emulating [wait=04] 0x00000004: inc ecx
(12:43:10) verbose in vstack: emulating [wait=05] 0x00000005: dec eax

It just works and distutils is now installed in the virtual environment. The pip install output does not mention distutils even once. I also installed pipdeptree and its output does not mention distutils either. I am confused.

cxiao commented 10 months ago

There is a possible explanation for the weirdness we're seeing here, where distutils is sometimes present inside a Python 3.12 virtual environment without a) having installed it explicitly, and b) without it being present in the output of pipdeptree.

According to the What’s New In Python 3.12 page, developers are encouraged to use the third-party setuptools package if they want to continue to use distutils:

PEP 632: Remove the distutils package. See the migration guide for advice replacing the APIs it provided. The third-party Setuptools package continues to provide distutils, if you still require it in Python 3.12 and beyond.

This behaviour manifests as follows, in a Python 3.12.0 virtual environment where setuptools is installed - distutils is importable as long as setuptools is imported first:

$ bin/python
Python 3.12.0 (main, Oct  2 2023, 00:00:00) [GCC 13.2.1 20230918 (Red Hat 13.2.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import distutils
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'distutils'
>>> import setuptools
>>> import distutils
>>>

This means that as long as setuptools has been imported first, importing any module that depends on distutils, such as unicorn, will just work. Therefore, in a Python 3.12.0 virtual environment where binary-refinery[all] has been installed, the following behaviour occurs:

$ bin/python
Python 3.12.0 (main, Oct  2 2023, 00:00:00) [GCC 13.2.1 20230918 (Red Hat 13.2.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import unicorn
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/cxiao/.local/pipx/venvs/binary-refinery-br-py3-12/lib64/python3.12/site-packages/unicorn/__init__.py", line 4, in <module>
    from .unicorn import Uc, uc_version, uc_arch_supported, version_bind, debug, UcError, __version__
  File "/home/cxiao/.local/pipx/venvs/binary-refinery-br-py3-12/lib64/python3.12/site-packages/unicorn/unicorn.py", line 5, in <module>
    import distutils.sysconfig
ModuleNotFoundError: No module named 'distutils'
>>> import setuptools
>>> import unicorn
>>>

huettenhain commented 10 months ago

My problem however, is that my Python test environment behaves slightly differently. After installing binary-refinery, this is what I get:

Python 3.12.0 (main, Oct 21 2023, 17:44:38) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import distutils
>>> distutils
<module 'distutils' (/tmp/venv/lib/python3.12/site-packages/setuptools/_distutils/__init__.py)>
>>>

So in my case, importing setuptools first is not necessary. It just works. This is likely also why the vstack unit just runs without issue: setuptools is part of refinery's build requirements and will therefore always be present. I just can't manage to reproduce the bug, and without that I don't know how to properly fix it.

jhhcs commented 10 months ago

I can indeed reproduce the issue with pipx while it works fine under pip. Very intriguing.

huettenhain commented 3 weeks ago

This issue is super stale and I am sorry about it. It's just such an annoying problem likely related to broken package management, I struggle to find the motivation. That said, I was given trustworthy advice to use icicle over unicorn and maybe when I do that, it will fix this problem magically.

binref / refinery