aqt install fail with a 7zip error

[yannou38]

We have a kind of weird CI using github action executing on runners located on a laptop, or on a azure VM. The runners share some installed programs, such as QT.

We have a github action to install QT on the laptop or the azure VM. The installation works properly on the laptop, but fail on the VM with the following log:

Notice:  A new release of pip available: 22.3.1 -> 23.2.1
Notice:  To update, run: python.exe -m pip install --upgrade pip
INFO    : aqtinstall(aqt) v3.1.6 on Python 3.11.1 [CPython MSC v.1934 64 bit (AMD64)]
ERROR   : Specified 7zip command executable does not exist: '7z'

Could you advise ? this error doesn't seem common from what i could search, but happen every time the github action execute on one of the azure VM runner.

The command executed is aqt install-qt windows desktop 6.3.0 win64_msvc2019_64 --outputdir foo

[yannou38]

Note: here is the install log for aqt on the azure VM:

Notice:  A new release of pip available: 22.3.1 -> 23.2.1
Notice:  To update, run: python.exe -m pip install --upgrade pip
Collecting aqtinstall
  Using cached aqtinstall-3.1.6-py3-none-any.whl (59 kB)
Collecting bs4
  Using cached bs4-0.0.1.tar.gz (1.1 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting defusedxml
  Using cached defusedxml-0.7.1-py2.py3-none-any.whl (25 kB)
Collecting humanize
  Using cached humanize-4.7.0-py3-none-any.whl (113 kB)
Collecting patch>=1.16
  Using cached patch-1.16.zip (12 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting py7zr>=0.20.2
  Using cached py7zr-0.20.5-py3-none-any.whl (66 kB)
Collecting requests>2.20.0
  Using cached requests-2.31.0-py3-none-any.whl (62 kB)
Collecting semantic-version
  Using cached semantic_version-2.10.0-py2.py3-none-any.whl (15 kB)
Collecting texttable
  Using cached texttable-1.6.7-py2.py3-none-any.whl (10 kB)
Collecting pycryptodomex>=3.6.6
  Using cached pycryptodomex-3.18.0-cp35-abi3-win_amd64.whl (1.7 MB)
Collecting pyzstd>=0.14.4
  Using cached pyzstd-0.15.9-cp311-cp311-win_amd64.whl (245 kB)
Collecting pyppmd<1.1.0,>=0.18.1
  Using cached pyppmd-1.0.0-cp311-cp311-win_amd64.whl (46 kB)
Collecting pybcj>=0.6.0
  Using cached pybcj-1.0.1-cp311-cp311-win_amd64.whl (24 kB)
Collecting multivolumefile>=0.2.3
  Using cached multivolumefile-0.2.3-py3-none-any.whl (17 kB)
Collecting brotli>=1.0.9
  Using cached Brotli-1.0.9-cp311-cp311-win_amd64.whl (333 kB)
Collecting inflate64>=0.3.1
  Using cached inflate64-0.3.1-cp311-cp311-win_amd64.whl (35 kB)
Collecting psutil
  Using cached psutil-5.9.5-cp36-abi3-win_amd64.whl (255 kB)
Collecting charset-normalizer<4,>=2
  Using cached charset_normalizer-3.2.0-cp311-cp311-win_amd64.whl (96 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting urllib3<3,>=1.21.1
  Using cached urllib3-2.0.4-py3-none-any.whl (123 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2023.7.22-py3-none-any.whl (158 kB)
Collecting beautifulsoup4
  Using cached beautifulsoup4-4.12.2-py3-none-any.whl (142kB)
Collecting soupsieve>1.2
  Using cached soupsieve-2.4.1-py3-none-any.whl (36 kB)
Installing collected packages: texttable, patch, brotli, urllib3, soupsieve, semantic-version, pyzstd, pyppmd, pycryptodomex, pybcj, psutil, multivolumefile, inflate64, idna, humanize, defusedxml, charset-normalizer, certifi, requests, py7zr, beautifulsoup4, bs4, aqtinstall
  DEPRECATION: patch is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for patch: started
  Running setup.py install for patch: finished with status 'done'
  DEPRECATION: bs4 is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for bs4: started
  Running setup.py install for bs4: finished with status 'done'
Successfully installed aqtinstall-3.1.6 beautifulsoup4-4.12.2 brotli-1.0.9 bs4-0.0.1 certifi-2023.7.22 charset-normalizer-3.2.0 defusedxml-0.7.1 humanize-4.7.0 idna-3.4 inflate64-0.3.1 multivolumefile-0.2.3 patch-1.16 psutil-5.9.5 py7zr-0.20.5 pybcj-1.0.1 pycryptodomex-3.18.0 pyppmd-1.0.0 pyzstd-0.15.9 requests-2.31.0 semantic-version-2.10.0 soupsieve-2.4.1 texttable-1.6.7 urllib3-2.0.4

[ddalcino]

Can you please show us the whole command you are using to install QT?

This error message only happens when you are using 7z to extract archives, instead of the internal py7zr tool. See the docs on the -E/--external flag: https://aqtinstall.readthedocs.io/en/latest/getting_started.html#external-7-zip-extractor

Simple fix: install 7z in your CI instance. Otherwise you can remove the -E flag.

[yannou38]

The command used to install qt is specified at the end of the issue text, and is the whole command. This same command is used successfully on the laptop, and 7zip is not installed on it either so I doubt we are accidentally using the external flag.

I'd also rather not have to install 7z manually on our CI instance if we can avoid it, but will keep the suggestion in mind.

[miurahr]

When aqt try import py7zr and failed, aqt fall back to use external command, which default is '7z'.. It is implemented in installer.py module. https://github.com/miurahr/aqtinstall/blob/8cbe51b3711062e2d5a7374d4ea7a8c6b4aea669/aqt/installer.py#L69-L74

here is a space we can leave a log message.

[yannou38]

Hello, and thanks for your reply. This was enlightening, and i searched why the module would fail to load.

It lead me to this error:

Traceback (most recent call last):
  File "C:\ghr\IOTSEC-gh-actions\1\_work\setup-qt\setup-qt\.venv\Lib\site-packages\py7zr\compressor.py", line 67, in <module>
    import brotli  # type: ignore  # noqa
    ^^^^^^^^^^^^^
  File "C:\ghr\IOTSEC-gh-actions\1\_work\setup-qt\setup-qt\.venv\Lib\site-packages\brotli.py", line 8, in <module>
    import _brotli
ImportError: DLL load failed while importing _brotli: The specified module could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen runpy>", line [189](https://github.schneider-electric.com/IOTSEC-gh-actions/setup-qt/actions/runs/453591/jobs/1078419#step:4:193), in _run_module_as_main
  File "<frozen runpy>", line 148, in _get_module_details
  File "<frozen runpy>", line 112, in _get_module_details
  File "C:\ghr\IOTSEC-gh-actions\1\_work\setup-qt\setup-qt\.venv\Lib\site-packages\py7zr\__init__.py", line 45, in <module>
    from py7zr.py7zr import ArchiveInfo, FileInfo, SevenZipFile, is_7zfile, pack_7zarchive, unpack_7zarchive
  File "C:\ghr\IOTSEC-gh-actions\1\_work\setup-qt\setup-qt\.venv\Lib\site-packages\py7zr\py7zr.py", line 43, in <module>
    from py7zr.archiveinfo import Folder, Header, SignatureHeader
  File "C:\ghr\IOTSEC-gh-actions\1\_work\setup-qt\setup-qt\.venv\Lib\site-packages\py7zr\archiveinfo.py", line 36, in <module>
    from py7zr.compressor import SevenZipCompressor, SevenZipDecompressor
  File "C:\ghr\IOTSEC-gh-actions\1\_work\setup-qt\setup-qt\.venv\Lib\site-packages\py7zr\compressor.py", line 69, in <module>
    import brotlicffi as brotli  # type: ignore  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'brotlicffi'

In my CI script, I installed brotlicffi manually, which did the trick and aqt works properly on the Azure VM now. I opened a new issue on py7zr to be looked at.

I'm unsure where this should be fixed, so i'll leave this issue open for now, but feel free to close it as needed, and once again thanks for your quick reply !

[Hish15]

Is there a valid reason to use 7z format instead of more classical format handled by python's shutil for instance zip?

[ddalcino]

Is there a valid reason to use 7z format instead of more classical format handled by python's shutil for instance zip?

Yes, there is. The Qt organization packages these files as 7z. We don’t get a choice in the matter.

Different compression algorithms have different compression ratios for different types of files. My guess is that the Qt developers chose 7z because it works well for the binaries they’re trying to compress. IIRC the ‘zip’ file format often doesn’t perform as well as many of the other formats; I can’t confirm this though.

[ddalcino]

here is a space we can leave a log message.

That’s a really good idea; it would help a lot in cases like this. What log level do you think it should be? I’m leaning towards ‘warning’.

[miurahr]

here is a space we can leave a log message.

That’s a really good idea; it would help a lot in cases like this. What log level do you think it should be? I’m leaning towards ‘warning’.

I prefer "warning" here.

[miurahr]

Because #705 is merged and released. Can I close here? @yannou38

[yannou38]

Yes, this issue can be closed