achille-martin / pyqt-crom

Create cross-platform apps (Android for now) using only Python and the Qt Framework (PyQt5 for now).
MIT License
25 stars 2 forks source link

Bug: pyqtdeploy-sysroot sip-module didn't create a sdist. #43

Open stone-cloud opened 1 month ago

stone-cloud commented 1 month ago

Hi,@achille-martin thank you for your amazing meticulous job and selfless open source. according to tutorial, I have an issue while implementing step 1.7. I couldn't find a solution to the relevant problem online, if you know how to solve it, please let me know and Thank you so much! The error message is as follows:

$ python3 build_app.py --pdt $PYQT_CROM_DIR/examples/demo/demo_project/config.pdt --jobs 1 --target android-64 --qmake $QT_DIR/android/bin/qmake --verbose

----- REVIEWING COMMAND-LINE ARGS -----

[INFO] The .pdt path received is: /home/gw/LYH/androidDepoly/pyqt-crom/examples/demo/demo_project/config.pdt
[INFO] The number of jobs received is: 1
[INFO] The qmake path received is: /home/gw/Qt/5.15.2/android/bin/qmake
[INFO] The request to reload the sysroot is: False
[INFO] The request to disable progress messages is: False
[INFO] The request to enable verbose progress messages is: True

----- INITIALISING AND COLLECTING VARIABLES -----

[INFO] Pdt directory location is: /home/gw/LYH/androidDepoly/pyqt-crom/examples/demo/demo_project. This is the reference directory.
[INFO] The sysroot path received is: /home/gw/LYH/androidDepoly/pyqt-crom/examples/demo/demo_project/sysroot.toml
[INFO] Setting application name to: DemoCrossPlatformApp
[INFO] The app entrypoint name received is: demo_app
[INFO] The app package path received is: /home/gw/LYH/androidDepoly/pyqt-crom/examples/demo/demo_project/demo_pkg
[INFO] The app release dir is set to: /home/gw/LYH/androidDepoly/pyqt-crom/examples/demo/demo_project/demo_pkg/../releases/2024_05_14-08_32_08

----- BUILDING TARGET SYSROOT -----

Verifying host architecture 'linux-64'...
Verifying target architecture 'android-64'...
zlib: verifying...
zlib: determining installed version from '/home/gw/Android/Sdk/ndk/21.4.7075529/sysroot/usr/include/zlib.h'.
zlib: verified v1.2.7.
Qt: verifying...
Running '/home/gw/Qt/5.15.2/android/bin/qmake -query'.
Qt: verified v5.15.2.
Python: verifying...
WARNING: Python: v3.8.19 is untested.
Python: verified v3.8.19.
SIP: verifying...
Running 'sip-module --version'.
SIP: verified v6.7.12.
PyQt: verifying...
PyQt: verified v5.15.10.
Deleting /home/gw/LYH/androidDepoly/pyqt-crom/examples/demo/demo_project/sysroot-android-64/build.
Creating /home/gw/LYH/androidDepoly/pyqt-crom/examples/demo/demo_project/sysroot-android-64/build.
SIP: installing component...
Running 'sip-module --sdist PyQt5.sip --abi-version 12'.
pyqtdeploy-sysroot: SIP: sip-module didn't create an sdist.

system information:

Distributor ID: Ubuntu
Description:    Ubuntu 20.04.6 LTS
Release:    20.04
Codename:   focal

Linux gw-System-Product-Name 5.15.0-105-generic #115~20.04.1-Ubuntu SMP Mon Apr 15 17:33:04 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
achille-martin commented 1 month ago

Hi @stone-cloud,

Thank you for your interest! And thanks for the debug prints, that is useful.

I am working on bug templates and a debugging tool for pyqt-crom, but it will be released in the next version, so for now we have to debug "manually".

First of all, I can see that you are using:

Now for your bug that might not be a problem, but for full support, I would recommend you to use:

That's quite restrictive for now because I can't keep track of thousands of specs!

Anyway, regarding your bug:

Observations

Research

Potential solutions

Here are some solutions to try out:

stone-cloud commented 1 month ago

Hi@achille-martin, First of all, I test Python-3.8.19 and Python-3.10.12 in the same Ubuntu20.04 system, and they all have this Bug. But I still can't be sure if it's unconcerned. I tried all the solutions, but none of them worked. The detailed information is as follows:

Try solutions

There might be something odd with PyQt5-sip module -> Can you run the command pip3 show PyQt5-sip (so I can know which version you are using -> it should be 12.13.0 according to requirements.txt).

Name: PyQt5-sip
Version: 12.13.0
Summary: The sip module support for PyQt5
Home-page: https://www.riverbankcomputing.com/software/sip/
Author: Riverbank Computing Limited
Author-email: info@riverbankcomputing.com
License: SIP
Location: /home/gw/anaconda3/envs/torch19/lib/python3.8/site-packages
Requires: 
Required-by: PyQt5
Name: PyQt5
Version: 5.15.10
Summary: Python bindings for the Qt cross platform application toolkit
Home-page: https://www.riverbankcomputing.com/software/pyqt/
Author: Riverbank Computing Limited
Author-email: info@riverbankcomputing.com
License: GPL v3
Location: /home/gw/anaconda3/envs/torch19/lib/python3.8/site-packages
Requires: PyQt5-Qt5, PyQt5-sip
Required-by: pyqtdeploy

Package                Version
---------------------- ------------
coloredlogs            15.0.1
dill                   0.3.8
flatbuffers            24.3.25
humanfriendly          10.0
mkl-fft                1.3.8
mkl-random             1.2.4
mkl-service            2.4.0
mpmath                 1.3.0
numpy                  1.24.3
onnx                   1.16.0
onnxconverter-common   1.13.0
onnxruntime-gpu        1.17.1
onnxruntime-tools      1.7.0
opencv-python-headless 4.9.0.80
packaging              23.2
pillow                 10.3.0
pip                    24.0
pipdeptree             2.18.1
ply                    3.11
protobuf               3.20.3
psutil                 5.9.8
py-cpuinfo             9.0.0
py3nvml                0.2.7
PyQt-builder           1.15.3
PyQt5                  5.15.10
PyQt5-Qt5              5.15.2
PyQt5-sip              12.13.0
pyqtdeploy             3.3.0
setuptools             69.5.1
sip                    6.7.12
sympy                  1.12
toml                   0.10.2
tomli                  2.0.1
torch                  1.9.0+cu111
torchaudio             0.9.0
torchvision            0.10.0+cu111
typing_extensions      4.11.0
wheel                  0.43.0
xmltodict              0.13.0

> Try to run the failed command in verbose mode: sip-module --sdist PyQt5.sip --abi-version 12 (but I am not sure that the flag --verbose will work) -> so observe the output in case errors are showing up

I tried, but no information was printed. šŸ˜«
> Move the [SIP] section to the bottom in your sysroot.toml and remove all the sysroot-android-64 generated folders to avoid caching issues and then rebuild the app

I tried, but it is not worked.
achille-martin commented 1 month ago

Ok, let's take it a step at a time then, since there are no debug prints associated to sip-module.

1) Run sip-module --sdist PyQt5.sip --abi-version 12 anywhere

It should download in the current directory the archive for PyQt5-sip matching with your installed version. So you should see a folder called PyQt5_sip-12.13.0.tar.gz in your current directory.

If you don't see it -> there is an issue with your sip or PyQt5-sip and I would recommend you pip3 uninstall and pip3 cache purge && pip3 install them back again.

If you see it -> then there might be an issue with your pyqtdeploy and we will need to add debug prints to $PYQT_CROM_DIR/venv/pyqt-crom-venv/lib/python3.10/site-packages/pyqtdeploy/sysroot/plugins/SIP.py.

stone-cloud commented 1 month ago

Amazingļ¼Base on your guidance, I found pyqt5_sip-12.13.0.tar.gz in my current directory. So I debug $PYQT_CROM_DIR/venv/pyqt-crom-venv/lib/python3.10/site-packages/pyqtdeploy/sysroot/plugins/SIP.py.

Debug

I found that glob can't recognize the pyqt5_sip-12.13.0.tar.gz due to a mismatch in the capitalization of the instructions archives = glob.glob(pattern)(https://www.riverbankcomputing.com/hg/pyqtdeploy/file/df680b956780/pyqtdeploy/sysroot/plugins/SIP.py). The file name is lowercase and the instructions contain uppercase.

Solutions

  1. First, What comes to my mind is to modify SIP module_name in pyqt-crom/examples/demo/demo_project/sysroot.toml. But it is not worked and print error.
  2. So I modify the code to archives = glob.glob(pattern.lower()) and it's working!

I'm not sure if this is a special case, such as the reason for the version number.

achille-martin commented 1 month ago

Glad you figured out the issue.

I suspect that the "lowercase" action is performed at some point in the sip package for some reason.

To investigate a bit further, can you:

If you get ['PyQt5.sip'], then the problem is likely later on in the code. Therefore, add debug prints in $PYQT_CROM_DIR/venv/pyqt-crom-venv/lib/python3.10/site-packages/sipbuild/module/module.pyto figure out when the lowercase operation happens (because it should not according tosip==6.7.12`).

If you get ['pyqt5.sip'], then the problem likely comes from the ArgumentParser (so argparse itself, which is built-in python3).

WARNINGS

1) It is not recommended to modify the site-packages directory directly

This is because changes cannot be tracked and it creates confusion when you want to change versions.

The recommended way is to use pip in "edit" mode -> https://pip.pypa.io/en/stable/topics/local-project-installs/#editable-installs

2) It is not recommended to modify the pyqtdeploy package directly

This is because of the licence which forces you to disclose all the changes you made to the package before a release of your app. If you are happy doing that, then you are fine.

stone-cloud commented 1 month ago

Hi @achille-martin ,

It's been a long time.

I tested the detailed output of '$PYQT_CROM_DIR/venv/pyqt-crom-venv/lib/python3.10/site-packages/sipbuild/module/module.py', and I found that when running the subprocess.run( [sys.executable, 'setup.py', '--quiet', 'sdist', '--dist-dir', '..']) command, the python package name, 'pyqt5_sip-12.13.0.tar.gz' is changed to lowercase letters.

I asked ChatGPT, and it told me that it might be because some systems are case-insensitive. And it suggested using all lowercase letters. Maybe it needs to consider the system version for testing in the next version.

achille-martin commented 1 month ago

@stone-cloud thanks for your debugging efforts!

I am a bit surprised by the fact that subprocess.run( [sys.executable, 'setup.py', '--quiet', 'sdist', '--dist-dir', '..']) modifies the module_name, I thought it would not affect it as it is installing from setup.py instructions already populated.

If you are happy to proceed, can you replace your $PYQT_CROM_DIR/venv/pyqt-crom-venv/lib/python3.10/site-packages/sipbuild/module/module.py file with the one I have attached:

module.txt

can you replace your $PYQT_CROM_DIR/venv/pyqt-crom-venv/lib/python3.10/site-packages/sipbuild/module/main.py file with the one I have attached:

main.txt.txt

I have added a lot of debug prints and I have added verbose tag to suspicious commands - make sure to save your original files (in site-packages) before replacing their content (so you can revert back to your version, alternatively you can reinstall sip via pip).

Then run sip-module --sdist PyQt5.sip --abi-version 12.

Then please send me the terminal outputs so we can have a look.

stone-cloud commented 1 month ago

The terminal outputs:

(androidDeploy) gw@gw-System-Product-Name:~/LYH/androidDepoly/QTgui$ sip-module --sdist PyQt5.sip --abi-version 12
main - ARGS: Namespace(abi_version='12', project=None, sdist=True, setup_cfg=None, sip_h=False, sip_rst=False, target_dir=None, sip_modules=['PyQt5.sip'])
module - SIP MODULE: PyQt5.sip
module - PROJECT: None
module - PROJECT 2: PyQt5_sip
_create_patches - SIP MODULE PARTS: ['PyQt5', 'sip']
_create_patches - SIP MODULE NAME: sip
_create_patches - SIP MODULE: PyQt5.sip
module - PATCHES: {'@SIP_MODULE_FQ_NAME@': 'PyQt5.sip', '@SIP_MODULE_PROJECT_NAME@': 'PyQt5_sip', '@SIP_MODULE_PACKAGE_NAME@': 'PyQt5', '@SIP_MODULE_VERSION@': '12.13.0', '@_SIP_MODULE_FQ_NAME@': 'PyQt5.sip', '@_SIP_MODULE_NAME@': 'sip', '@_SIP_MODULE_SHARED@': '1', '@_SIP_MODULE_ENTRY@': 'PyInit_sip', '@_SIP_MODULE_LEGACY@': '1', '@_SIP_VERSION@': '0x6070c', '@_SIP_VERSION_STR@': '6.7.12'}
module - SDIST DIR: PyQt5_sip-12.13.0
module - SDIST DIR 2: PyQt5_sip-12.13.0
_create_sdist - SDIST DIR: PyQt5_sip-12.13.0
_create_sdist - SDIST DIR 2: PyQt5_sip-12.13.0
_create_sdist - MODULE SOURCE DIR: /home/gw/anaconda3/envs/androidDeploy/lib/python3.10/site-packages/sipbuild/module/source/12
_create_sdist - SETUP CONFIG: None
_create_sdist - NAME FROM SOURCE DIR: MANIFEST.in
_create_sdist - NAME FROM SOURCE DIR: objmap.c
_create_sdist - NAME FROM SOURCE DIR: sip_array.h
_create_sdist - NAME FROM SOURCE DIR: siplib.c
_create_sdist - NAME FROM SOURCE DIR: sipint.h
_create_sdist - NAME FROM SOURCE DIR: LICENSE
_create_sdist - NAME FROM SOURCE DIR: threads.c
_create_sdist - NAME FROM SOURCE DIR: sip.rst.in
_create_sdist - NAME FROM SOURCE DIR: sip_array.c
_create_sdist - NAME FROM SOURCE DIR: setup.py.in
_install_source_file - module_source_dir: /home/gw/anaconda3/envs/androidDeploy/lib/python3.10/site-packages/sipbuild/module/source/12
name: setup.py
target_dir: PyQt5_sip-12.13.0
_install_file - NAME OUT: PyQt5_sip-12.13.0/setup.py
PATCHES: {'@SIP_MODULE_FQ_NAME@': 'PyQt5.sip', '@SIP_MODULE_PROJECT_NAME@': 'PyQt5_sip', '@SIP_MODULE_PACKAGE_NAME@': 'PyQt5', '@SIP_MODULE_VERSION@': '12.13.0', '@_SIP_MODULE_FQ_NAME@': 'PyQt5.sip', '@_SIP_MODULE_NAME@': 'sip', '@_SIP_MODULE_SHARED@': '1', '@_SIP_MODULE_ENTRY@': 'PyInit_sip', '@_SIP_MODULE_LEGACY@': '1', '@_SIP_VERSION@': '0x6070c', '@_SIP_VERSION_STR@': '6.7.12'}
_create_sdist - NAME FROM SOURCE DIR: bool.cpp
_create_sdist - NAME FROM SOURCE DIR: setup.cfg.in
_create_sdist - NAME FROM SOURCE DIR: LICENSE-GPL3
_create_sdist - NAME FROM SOURCE DIR: int_convertors.c
_create_sdist - NAME FROM SOURCE DIR: voidptr.c
_create_sdist - NAME FROM SOURCE DIR: apiversions.c
_create_sdist - NAME FROM SOURCE DIR: LICENSE-GPL2
_create_sdist - NAME FROM SOURCE DIR: sip.pyi
_create_sdist - NAME FROM SOURCE DIR: pyproject.toml
_create_sdist - NAME FROM SOURCE DIR: sip.h.in
_install_source_file - module_source_dir: /home/gw/anaconda3/envs/androidDeploy/lib/python3.10/site-packages/sipbuild/module/source/12
name: sip.h
target_dir: PyQt5_sip-12.13.0
_install_file - NAME OUT: PyQt5_sip-12.13.0/sip.h
PATCHES: {'@SIP_MODULE_FQ_NAME@': 'PyQt5.sip', '@SIP_MODULE_PROJECT_NAME@': 'PyQt5_sip', '@SIP_MODULE_PACKAGE_NAME@': 'PyQt5', '@SIP_MODULE_VERSION@': '12.13.0', '@_SIP_MODULE_FQ_NAME@': 'PyQt5.sip', '@_SIP_MODULE_NAME@': 'sip', '@_SIP_MODULE_SHARED@': '1', '@_SIP_MODULE_ENTRY@': 'PyInit_sip', '@_SIP_MODULE_LEGACY@': '1', '@_SIP_VERSION@': '0x6070c', '@_SIP_VERSION_STR@': '6.7.12'}
_create_sdist - NAME FROM SOURCE DIR: qtlib.c
_create_sdist - NAME FROM SOURCE DIR: descriptors.c
_create_sdist - NAME FROM SOURCE DIR: README.in
_install_source_file - module_source_dir: /home/gw/anaconda3/envs/androidDeploy/lib/python3.10/site-packages/sipbuild/module/source/12
name: README
target_dir: PyQt5_sip-12.13.0
_install_file - NAME OUT: PyQt5_sip-12.13.0/README
PATCHES: {'@SIP_MODULE_FQ_NAME@': 'PyQt5.sip', '@SIP_MODULE_PROJECT_NAME@': 'PyQt5_sip', '@SIP_MODULE_PACKAGE_NAME@': 'PyQt5', '@SIP_MODULE_VERSION@': '12.13.0', '@_SIP_MODULE_FQ_NAME@': 'PyQt5.sip', '@_SIP_MODULE_NAME@': 'sip', '@_SIP_MODULE_SHARED@': '1', '@_SIP_MODULE_ENTRY@': 'PyInit_sip', '@_SIP_MODULE_LEGACY@': '1', '@_SIP_VERSION@': '0x6070c', '@_SIP_VERSION_STR@': '6.7.12'}
_create_sdist - SETUP CONFIG 2: /home/gw/anaconda3/envs/androidDeploy/lib/python3.10/site-packages/sipbuild/module/source/12/setup.cfg.in
_install_file - NAME OUT: PyQt5_sip-12.13.0/setup.cfg
PATCHES: {'@SIP_MODULE_FQ_NAME@': 'PyQt5.sip', '@SIP_MODULE_PROJECT_NAME@': 'PyQt5_sip', '@SIP_MODULE_PACKAGE_NAME@': 'PyQt5', '@SIP_MODULE_VERSION@': '12.13.0', '@_SIP_MODULE_FQ_NAME@': 'PyQt5.sip', '@_SIP_MODULE_NAME@': 'sip', '@_SIP_MODULE_SHARED@': '1', '@_SIP_MODULE_ENTRY@': 'PyInit_sip', '@_SIP_MODULE_LEGACY@': '1', '@_SIP_VERSION@': '0x6070c', '@_SIP_VERSION_STR@': '6.7.12'}
_create_sdist - SAVED CWD: /home/gw/LYH/androidDepoly/QTgui
_create_sdist - SDIST DIR 2: PyQt5_sip-12.13.0
_create_sdist - SYS EXEC: /home/gw/anaconda3/envs/androidDeploy/bin/python
running sdist
running egg_info
creating PyQt5_sip.egg-info
writing PyQt5_sip.egg-info/PKG-INFO
writing dependency_links to PyQt5_sip.egg-info/dependency_links.txt
writing top-level names to PyQt5_sip.egg-info/top_level.txt
writing manifest file 'PyQt5_sip.egg-info/SOURCES.txt'
reading manifest file 'PyQt5_sip.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
adding license file 'LICENSE-GPL2'
adding license file 'LICENSE-GPL3'
writing manifest file 'PyQt5_sip.egg-info/SOURCES.txt'
running check
creating pyqt5_sip-12.13.0
creating pyqt5_sip-12.13.0/PyQt5_sip.egg-info
copying files to pyqt5_sip-12.13.0...
copying LICENSE -> pyqt5_sip-12.13.0
copying LICENSE-GPL2 -> pyqt5_sip-12.13.0
copying LICENSE-GPL3 -> pyqt5_sip-12.13.0
copying MANIFEST.in -> pyqt5_sip-12.13.0
copying README -> pyqt5_sip-12.13.0
copying apiversions.c -> pyqt5_sip-12.13.0
copying bool.cpp -> pyqt5_sip-12.13.0
copying descriptors.c -> pyqt5_sip-12.13.0
copying int_convertors.c -> pyqt5_sip-12.13.0
copying objmap.c -> pyqt5_sip-12.13.0
copying pyproject.toml -> pyqt5_sip-12.13.0
copying qtlib.c -> pyqt5_sip-12.13.0
copying setup.cfg -> pyqt5_sip-12.13.0
copying setup.py -> pyqt5_sip-12.13.0
copying sip.h -> pyqt5_sip-12.13.0
copying sip_array.c -> pyqt5_sip-12.13.0
copying sip_array.h -> pyqt5_sip-12.13.0
copying sipint.h -> pyqt5_sip-12.13.0
copying siplib.c -> pyqt5_sip-12.13.0
copying threads.c -> pyqt5_sip-12.13.0
copying voidptr.c -> pyqt5_sip-12.13.0
copying PyQt5_sip.egg-info/PKG-INFO -> pyqt5_sip-12.13.0/PyQt5_sip.egg-info
copying PyQt5_sip.egg-info/SOURCES.txt -> pyqt5_sip-12.13.0/PyQt5_sip.egg-info
copying PyQt5_sip.egg-info/dependency_links.txt -> pyqt5_sip-12.13.0/PyQt5_sip.egg-info
copying PyQt5_sip.egg-info/top_level.txt -> pyqt5_sip-12.13.0/PyQt5_sip.egg-info
copying PyQt5_sip.egg-info/SOURCES.txt -> pyqt5_sip-12.13.0/PyQt5_sip.egg-info
Writing pyqt5_sip-12.13.0/setup.cfg
Creating tar archive
removing 'pyqt5_sip-12.13.0' (and everything under it)
_create_sdist - SAVED CWD 2: /home/gw/LYH/androidDepoly/QTgui
_create_sdist - SDIST DIR 3: PyQt5_sip-12.13.0

and in my root, I get PyQt5_sip-12.13.0 fold and 'pyqt5_sip-12.13.0.tar.gz' file.

achille-martin commented 1 month ago

Ok I can confirm that you are right about the command setup.py messing things up.

I dug a bit more into the setup template and I have added a few more debug prints to module.py:

module.txt

Again, if you have time, replace module.py with my extra verbose code and run sip-module --sdist PyQt5.sip --abi-version 12 and we'll inspect the output.

This time, the output will tell us from which python function the problem emerges (it might be the replace).

If this round comes back inconclusive, I'll close the issue and mark it as "temporarily fixed" with your previous fix.

stone-cloud commented 1 month ago

All right! The print log information is in log file:

log.txt

If you locate where the problem occurred, please let me know.

achille-martin commented 1 month ago

Ah! I found it... I was looking at the wrong place...

Details of the issue

The "issue" comes from setuptools v69.3.0 Changelog infos here:

The changes have been made with this chronology:

Context of the issue

Solutions

1) Constrain setuptools to be < v69.3.0 in requirements.txt -> quite restrictive but works for now 2) Modify self.module_name to be canonical in pyqtdeploy/pyqtdeploy/sysroot/plugins/SIP.py so that it can find the archive downloaded (the tar.gz) -> need to check with Phil 3) Modify the value of module_name to pyqt5.sip in sysroot.toml under [SIP]

Thanks for your debugging support!

achille-martin commented 1 month ago

TL;DR: for now, use pip install setuptools==69.2.0

stone-cloud commented 1 month ago

That's great! We can continue doing this great work. :wink: