Closed LuisHenri closed 5 months ago
Hi @LuisHenri, in the Wheel submitted to the PyPI we can see that the pkg_resources/extern/__init__.py
file exists: https://inspector.pypi.io/project/setuptools/70.0.0/packages/de/88/70c5767a0e43eb4451c2200f07d042a4bcd7639276003a9c54a68cfcc1f8/setuptools-70.0.0-py3-none-any.whl/pkg_resources/extern/__init__.py. It is a bit weird that we are getting the error ModuleNotFoundError: No module named 'pkg_resources.extern'
(considering that the file is there).
So I am not sure why this error is happening. Would it be possible to create a reproducer without packing the main? I am not sure how PyArmor/PyInstaller works internally and if that is somehow related to this problem...
I have the same error this morning at https://github.com/Chia-Network/chia-blockchain/actions/runs/9175546125/job/25229722015?pr=16898. PyInstaller has to pick what files to include in it's output executable and I'm guessing it needs adapted to a change in setuptools v70. But, I'm just starting to look now.
@LuisHenri, @altendky, when I try to run the reproducer in a controlled environment, I don't see the error reported:
> docker run --rm -it python:3.8-windowsservercore-1809 powershell
mkdir $env:TEMP\myproj
cd $env:TEMP\myproj
new-item main.py
@"
print('hello world')
"@ | add-content -Path main.py
python -m pip install -U pip
python -m pip install -U setuptools==70.0.0
python -m pip install -U pyarmor==7.* pyinstaller[encryption]==4.*
python -m pip list
# Package Version
# ------------------------- --------
# altgraph 0.17.4
# importlib_metadata 7.1.0
# packaging 24.0
# pefile 2023.2.7
# pip 24.0
# pyarmor 7.7.4
# pyinstaller 4.10
# pyinstaller-hooks-contrib 2024.6
# pywin32-ctypes 0.2.2
# setuptools 70.0.0
# tinyaes 1.1.0
# wheel 0.36.2
# zipp 3.18.2
pyarmor pack main.py
# ...
# INFO PyArmor Trial Version 7.7.4
# ...
# 226 INFO: PyInstaller: 4.10
# 226 INFO: Python: 3.8.10
# 241 INFO: Platform: Windows-10-10.0.17763-SP0
# ...
# INFO Final output path: dist
# INFO Pack obfuscated scripts successfully.
.\dist\main\main.exe
# hello world
(I don't have access to a machine with Windows 11 to try the reproducer).
I'm using PyInstaller 6.6.0, fwiw, and no encryption etc. I think I'm also only having an issue on Windows (which you tested). I'll let you know when I figure anything useful out.
I'm using PyInstaller 6.6.0, fwiw, and no encryption etc.
I have also run the given reproducer on PyInstaller 6.6.0 with no encryption:
> docker run --rm -it python:3.8-windowsservercore-1809 powershell
mkdir $env:TEMP\myproj
cd $env:TEMP\myproj
new-item main.py
@"
import math
print(f'hello world: {math.pi=}')
"@ | add-content -Path main.py
python -m pip install -U pip
python -m pip install -U setuptools==70.0.0
python -m pip install -U pyarmor==7.* pyinstaller==6.6.0
python -m pip list
# Package Version
# ------------------------- --------
# altgraph 0.17.4
# importlib_metadata 7.1.0
# packaging 24.0
# pefile 2023.2.7
# pip 24.0
# pyarmor 7.7.4
# pyinstaller 6.6.0
# pyinstaller-hooks-contrib 2024.6
# pywin32-ctypes 0.2.2
# setuptools 70.0.0
# wheel 0.36.2
# zipp 3.18.2
pyarmor pack main.py
# ...
# INFO PyArmor Trial Version 7.7.4
# INFO Python 3.8.10
# INFO Target platforms: Native
# ...
# 273 INFO: PyInstaller: 6.6.0, contrib hooks: 2024.6
# 273 INFO: Python: 3.8.10
# 273 INFO: Platform: Windows-10-10.0.17763-SP0
# ...
# INFO Final output path: dist
# INFO Pack obfuscated scripts successfully.
.\dist\main\main.exe
# hello world
The program seems to work without the exception...
Maybe the description of the reproducer need a bit more detail?
@abravalheri
So, I tested somethings one more time here and adding pkg_resources.extern
to PyInstaller's Hidden Imports fixes the issue.
As @altendky said, it might actually be something that PyInstaller needs to fix upon the changes on SetupTools. Although I wonder what actually changed on SetupTools that made it break on PyInstaller side
@abravalheri, I notice you are running Windows in docker. Might affect the outcome. I'm in Linux and haven't run that same example yet, just been trying to explore my own case here. Waiting on CI. *twiddles thumbs*
And...
https://github.com/Chia-Network/chia-blockchain/actions/runs/9178294773/job/25237858145?pr=18049 shows a pass with setuptools 804ccd2ff7d01d97b5f23279508724ddfb1bb094 while https://github.com/Chia-Network/chia-blockchain/actions/runs/9178307533/job/25237903410?pr=18050 shows a failure with setuptools e9995828311c5e0c843622ca2be85e7f09f1ff0d. Just to narrow down the change to the latter commit.
Although I wonder what actually changed on SetupTools that made it break on PyInstaller side
Difficult to guess...
There were some changes in the imports in pkg_resources
: https://github.com/pypa/setuptools/compare/v69.5.1..v70.0.0.
@abravalheri, I notice you are running Windows in docker. Might affect the outcome. I'm in Linux and haven't run that same example yet, just been trying to explore my own case here. Waiting on CI.
*twiddles thumbs*
And...
https://github.com/Chia-Network/chia-blockchain/actions/runs/9178294773/job/25237858145?pr=18049 shows a pass with setuptools 804ccd2 while https://github.com/Chia-Network/chia-blockchain/actions/runs/9178307533/job/25237903410?pr=18050 shows a failure with setuptools e999582. Just to narrow down the change to the latter commit.
So in that commit there was a change in the way the imports are done:
- from pkg_resources.extern import platformdirs
- from pkg_resources.extern import packaging
-
- __import__('pkg_resources.extern.packaging.version')
- __import__('pkg_resources.extern.packaging.specifiers')
- __import__('pkg_resources.extern.packaging.requirements')
- __import__('pkg_resources.extern.packaging.markers')
- __import__('pkg_resources.extern.packaging.utils')
+ from pkg_resources.extern.packaging import markers as _packaging_markers
+ from pkg_resources.extern.packaging import requirements as _packaging_requirements
+ from pkg_resources.extern.packaging import utils as _packaging_utils
+ from pkg_resources.extern.packaging import version as _packaging_version
+ from pkg_resources.extern.platformdirs import user_cache_dir
But that is not something that changes the way things work in runtime... Maybe the heuristics PyInstaller uses are too specific?
PyInstaller having to keep up with changes "like this" is indeed common. FWIW, I went and looked for an issue report against PyInstaller before I cam over here. :]
I'll file an issue over there, if nobody beats me to it.
@altendky I was just going to open an Issue there. But you can head it :P
I was just looking at a hook that PyInstaller has specifically for pkg_resources
already inside it: https://github.com/pyinstaller/pyinstaller/blob/develop/PyInstaller/hooks/hook-pkg_resources.py
Probably they'll need to update it.
PyInstaller having to keep up with changes "like this" is indeed common
Crazy @_@ I wonder if they'll just fix the most recent version (v6) or also some older ones (I use v4, f. ex.)
Using __import__
instead of import
is enough to break PyInstaller's dependency scanning. It can't see dynamic imports (even on constant strings).
The interesting thing about that is that https://github.com/pypa/setuptools/commit/e9995828311c5e0c843622ca2be85e7f09f1ff0d went the other direction. But that commit also went and switched to as _*
imports. Anyways, this is likely trivial for Hartmut to fix once they look at the commit.
Thank you very much for having a deep look at this and contacting PyInstaller.
I suppose that this is going to be handled in https://github.com/pyinstaller/pyinstaller/issues/8554, with no further action in setuptools.
I will leave it open for a while so users can easily find it.
Anyways, this is likely trivial for Hartmut to fix once they look at the commit.
Harmut's tolerance ran out a long time ago.
We'll fix this on our end. Feel free to close this off as PyInstaller's problem.
Should be fixed now. pip install -U pyinstaller
then use the --clean
flag next time you run pyinstaller
.
Works for me. Thanks again here too. :]
Should be fixed now.
pip install -U pyinstaller
then use the--clean
flag next time you runpyinstaller
.
For clarity for future readers: the changes introduced in v70.0.0 of setuptools have been accounted for in v6.7.0 of pyinstaller onwards (released May 21st 2024).
https://github.com/pyinstaller/pyinstaller/releases/tag/v6.7.0 https://pyinstaller.org/en/v6.7.0/CHANGES.html#hooks https://github.com/pyinstaller/pyinstaller/pull/8555
setuptools version
setuptools==70.0.0
Python version
3.8.10
OS
Windows 11
Additional environment information
No response
Description
I use PyInstaller and PyArmor to obfuscate my code and create an Executable out of it. On a pipeline of mine, an error started to throw today:
Inside this Pipeline I run
python -m pip install -U setuptools
. I.e., it uses the latest version of it.I tested it locally and it happens with version 70.0.0 but not with version 69.5.1
Expected behavior
Obfuscated Executable from PyArmor and PyInstaller is created.
How to Reproduce
pip install tendo==0.2.*
(just for the matter of having some import on the main.py file) (for some reason, it works with tendo>=0.3.0)Create a test.py
pyarmor pack .\test.py -x "--exclude .venv"
dist/test/
Output