Possible regression pyinstaller 5.10.0 in combination with staticx?

[erikbosch]

I am working on the Eclipse KUKSA.val project and I experienced a regression when using the latest 5.10.0 pyinstaller. It occurs for a Dockerfile where first pyinstaller is called, then staticx. The error occurs when staticx is run, but with some references to pyinstaller.py. If specifying that pyinstaller 5.9.0 shall be used there is no problem. I must admit that I have limited knowledge of the pyinstaller and staticx relationship, so any help in analyzing the error would be appreciated.

Using pyinstaller 5.10.0

13 [builder  8/14] RUN pyinstaller --collect-data kuksa_certificates --hidden-import can.interfaces.socketcan --clean -F -s dbcfeeder.py
#13 0.657 150 INFO: PyInstaller: 5.10.0
#13 0.657 150 INFO: Python: 3.9.14
#13 0.660 152 INFO: Platform: Linux-5.10.0-20-amd64-x86_64-with-glibc2.31
#13 0.660 153 INFO: wrote /dbcfeeder.spec
// Some lines removed
#13 DONE 28.5s

#14 [builder  9/14] WORKDIR /dist
#14 DONE 0.0s

#15 [builder 10/14] RUN staticx dbcfeeder dbcfeeder-exe
#15 1.013 Traceback (most recent call last):
#15 1.013   File "/opt/venv/bin/staticx", line 8, in <module>
#15 1.013     sys.exit(main())
#15 1.013   File "/opt/venv/lib/python3.9/site-packages/staticx/__main__.py", line 49, in main
#15 1.013     generate(args.prog, args.output,
#15 1.013   File "/opt/venv/lib/python3.9/site-packages/staticx/api.py", line 328, in generate
#15 1.013     gen.generate(output=output)
#15 1.013   File "/opt/venv/lib/python3.9/site-packages/staticx/api.py", line 134, in generate
#15 1.013     run_hooks(self)
#15 1.013   File "/opt/venv/lib/python3.9/site-packages/staticx/hooks/__init__.py", line 12, in run_hooks
#15 1.013     hook(sx)
#15 1.013   File "/opt/venv/lib/python3.9/site-packages/staticx/hooks/pyinstaller.py", line 27, in process_pyinstaller_archive
#15 1.014     h.process()
#15 1.014   File "/opt/venv/lib/python3.9/site-packages/staticx/hooks/pyinstaller.py", line 47, in process
#15 1.014     binaries = self._extract_binaries()
#15 1.014   File "/opt/venv/lib/python3.9/site-packages/staticx/hooks/pyinstaller.py", line 63, in _extract_binaries
#15 1.014     for n, item in enumerate(self.pyi_ar.toc.data):
#15 1.014 AttributeError: 'dict' object has no attribute 'data'
#15 ERROR: process "/bin/sh -c staticx dbcfeeder dbcfeeder-exe" did not complete successfully: exit code: 1
------
 > [builder 10/14] RUN staticx dbcfeeder dbcfeeder-exe:
#15 1.013     run_hooks(self)
#15 1.013   File "/opt/venv/lib/python3.9/site-packages/staticx/hooks/__init__.py", line 12, in run_hooks
#15 1.013     hook(sx)
#15 1.013   File "/opt/venv/lib/python3.9/site-packages/staticx/hooks/pyinstaller.py", line 27, in process_pyinstaller_archive
#15 1.014     h.process()
#15 1.014   File "/opt/venv/lib/python3.9/site-packages/staticx/hooks/pyinstaller.py", line 47, in process
#15 1.014     binaries = self._extract_binaries()
#15 1.014   File "/opt/venv/lib/python3.9/site-packages/staticx/hooks/pyinstaller.py", line 63, in _extract_binaries
#15 1.014     for n, item in enumerate(self.pyi_ar.toc.data):
#15 1.014 AttributeError: 'dict' object has no attribute 'data'

Using pyinstaller 5.9.0

#15 [builder  8/14] RUN pyinstaller --collect-data kuksa_certificates --hidden-import can.interfaces.socketcan --clean -F -s dbcfeeder.py
#15 0.662 114 INFO: PyInstaller: 5.9.0
#15 0.662 114 INFO: Python: 3.9.14
#15 0.666 117 INFO: Platform: Linux-5.10.0-20-amd64-x86_64-with-glibc2.31
#15 0.667 118 INFO: wrote /dbcfeeder.spec
// Some lines removed
#15 DONE 27.5s

#16 [builder  9/14] WORKDIR /dist
#16 DONE 0.0s

#17 [builder 10/14] RUN staticx dbcfeeder dbcfeeder-exe
#17 DONE 20.3s

[rokm]

Looks like staticx is using PyInstaller's archive reader classes, which have been changed in #7518. As those classes are internal to PyInstaller and not intended for outside use, the corresponding breakage, while unfortunate, is not a regression from my perspective.

[erikbosch]

Thanks for the analysis @rokm. I created an issue in staticx (https://github.com/JonathonReinhart/staticx/issues/235) so they can decide how they want to handle it.

[JonathonReinhart]

I agree with @rokm, this is an issue with staticx. It is, as you said, an unfortunate consequence of using the internal APIs, but I had little choice.

It should be straightforward enough to detect the change from #7518 at runtime and use the new interface.

We can close this issue and track in JonathonReinhart/staticx#235.