achille-martin / pyqt-crom

Create cross-platform apps (Android for now) using only Python and the Qt Framework (PyQt5 for now).
MIT License
29 stars 2 forks source link

Bug: Cannot add Python packages from site-packages directory of my Python interpreter to the .pdt of pyqtdeploy 3.3.0 #32

Open jerryfat opened 7 months ago

jerryfat commented 7 months ago

Hi Achille Martin: Thank you for the readme and pictures, I was successfull building the demo app and running on emulator and tablet Your demo app worked on emulator and phone I have Great Instructions !

Looked like in previous 2.5.1 pyqtdeploy looks like user was able to add the site-packages directory and then include the packages. I tried using the 2.5.1 pyqtdeploy and it cant open the pdy and find sysroot etc etc

when I build using 3.3.0 pyqtdelpoy my app code builds but apk app crashes , I tried to build my app but I dont see "Other packages" and installed packages like yaml and lxlm site-packages using pip install in venv or not venv too my app runs on linux ok, but not emulator or tablet I am assumming my app crashes on android since it cant find the imported packages that should have been frozen etc

can I add or scan directories for installed Other packages in your new 3.3.0 pyqtdeploy version ?

i didnt see how to add a sysroot directory like in 2.5.1 that looked and worked nice I was trying to use the 2.5.1 ver and (wouldnt build) but that was so nice in 2.5.1 , check marking the packages to include in build ,

In your readme maybe include another picture of the Other packages and add package if that feature works. If there is a way to add packages manually let me know or workarounds I appreciate your work very much, so nice the demo and readme work Thanks so very much in advance, Jerry Fat
my project on github: https://github.com/jerryfat/FIX-Gateway/blob/master/README-MAVPX4.md

achille-martin commented 7 months ago

Thank you for your interest @jerryfat!

Let's try to unpack what you are asking about and let's figure out the answer together.

I haven't created a template to write issues yet because I didn't actually expect this project to attract developers and programmers. However, since it seems that you have found some interest in the project, I'll bump this task up!

For the title, I would focus on: <keyword>: <problem / topic>

<keyword> could be:

In your case, it seems that a good title would be: Bug: cannot add Python packages from site-packages directory of my Python interpreter to the .pdt of pyqtdeploy 3.3.0

Moving on to the content of the bug/request, I would break it down into:

Bug

Description

Objective: What is the expected behaviour / What are you trying to achieve

Observations: Describe the observed unexpected behaviour, adding as many details as possible (including screenshots, when needed).

Version: [Component or package version(s) affected by this issue; it can be identified by a tag, or a commit SHA](Link to the tag/commit)

Reporter: Name Surname \<email> of the person reporting the bug.

Linked Issues: [Name of the linked issue)](Link to related issue)

Replication

Provide a comprehensive list of step-by-step instructions to reproduce the issue.

Explanation

After investigating the reported bug, give a brief explanation of the reason which is believed to cause the issue.

Proposed solution

Describe the proposed solution to address the bug.

Notes:

Additional notes about the proposed solution.

achille-martin commented 7 months ago

Your issue actually gathers all the elements that I have listed above, which is great.

Bug: Cannot add Python packages from site-packages directory of my Python interpreter to the .pdt of pyqtdeploy 3.3.0

Description

Objective: You want to add yaml and lxlm to your Android app using the .pdt

Observations:

Version: pyqtdeploy 3.3.0; yaml version?; lxlm version?

Reporter: @jerryfat

Linked Issues: None

Replication

Do you have an example PyQt5 script with yaml to share here? As well as the steps you took to install yaml in your venv?

Explanation

Initial research

Existing resources

Going further

Proposed solution

We will update this section and then add the whole bug / solution note to the PyQt-CroM troubleshooting note (if needed).

Notes:

Personal notes: Your project looks great, I had a very brief experience with the Mavlinks during a competition at Uni.

achille-martin commented 7 months ago

As a bonus, I have added your feedback to the specific discussion thread if you want to complete it.

jerryfat commented 7 months ago

Hi Achille: Thank you for creating detailed emails for "issues" message I left on your github Do you want me to do anything more, I can test or give you opinions lol I appreciate your patience in detailing such a complicated issue in your emails in response to my newbie questions, You "hit the nail on the head" as they say or "Bullseye"

Any pics are so helpful but difficult to create, maybe a flowchart of the the overall process and or the vars you use once it works it so cool you got all this working and the readme too thanks for a very good and concise document you write very well .

I stumbled upon your github by chance I have tried the necesitas method of pyqt into android and it was really clumsy and didnt really seem to work right

I can test or give you opinions lol or add issues or whatever you want I have some time if I can help you somehow...

btw from your readme somehow the printf "" of vars into .bashrc is in a loop on my machine I had to stop the command and hand remove the entries at end of .bashrc I dont know why its doing that

Do you want me to see why that >> .bashrc command in your readme is doing that here ? it looks correct in readme but didnt work here in xterm for me ubuntu 20.1 mint only problem I had with your readme

Can I use your 2.5.1 version to build with apps with "Other site-packages" ?

the sysroot you create is really cool concept, trying to understand what you are doing

and I dont yet know how does sysroot get set there in 2.5.1 or works ? i see the change from xml to python config file (?) python style of config vars from 2.5.1 to 3.3.0 and I see why you changed from .pdy to .pdt suffix, good idea

Is there a link or description somewhere to older working build and version 2.5.1 that I can try with other packages to create android apk? Is there a readme or working pyqtdeploy using the 2.5.1 ? using which Python ? that can be used ?

and are there any manual ways I can add Other packages to your 3.3.0 ? or is that feature still needs to be added ? it was in 2.5.1 and works great , nice interface in 2.5.1

Finest Regards, and thanks again

achille-martin commented 7 months ago
jerryfat commented 7 months ago

Hi Achille: Appreciate pyqtdeploy3.3.0 document of the 'Other packages' or 'site-packages' for my issue how did you make such a nice doc btw ? very nice and captures this info well

Appreciate all those links at end of doc, and how I could add a package or module to the pyqtdeply3.3.0 pdt , did not know you could do that I will also try this and let you know if it works here, but this getting kinda "outside my wheelhouse" lol as they say

I am assumming (!) yaml or lxml or any other site-packages package should do same thing as a test for the python binary from the pdt ?, so any imported module (from site-packages or from pip install)? of venv target python binary ? (from the binary python attached venv sysroot ? my hope is issue should somewhow automagically (for me) be detected and the site packages appear in pyqtdeploy3.3.0 Other packages tab

I am guessing that by specifying the python version in pyqtdeploy3.3.0 and the .pdt, the build process downloads and builds a binary which then needs all the site packages as static(freeze) and then linked into app , way over my head right now.

also guessing so by adding the packages from a wheel in the pdt they will get built as binaries and linked into final app binary

I was going to add these test import yaml or lxml to the demo project in the instructions that builds ok and will add a line or two in app code that calls will send to you I dont know how to look under the hood of the pyqtdeploy app or pyQt or python, still a mystery and learning

This is getting way over my understanding of the magic you guys like Phil Thompson make I had no idea the complexity and now struggling to understand and make stuff work on apk

I sent an email to Qt inf at riverbank email aaddress, only after trying to create a new login for their forums. which didnt seem to work for me. I hope I didnt anger someone like Phil Thompson , my new hero.

I could not reproduce 'my problem' on command line in .bashrc from your most excellent instructions, I now have no idea why my .bashrc filled up with thousands of those text lines. Finest Regards, and Sincerely,

achille-martin commented 7 months ago
jerryfat commented 7 months ago

Hi Achille: I wrote an app that checks for the .whl for imports in a .py file I have attached the code i will git hub it soon if no -f on command line it uses the app imports if -f filename.py it checks filename.py for imports and gets whl files app tries to get .whl using pip download and puts into ./wheels directory it then creates the sysroot.toml style formatted text for inclusion into the pyqtdeploy3.3.0 I will try a few demos to see if this brings in the wheels for each import and dependencies packages python 3.10

list-modules-imported.py


Cf. file list-modules-imported-v1.txt

list-modules-imported-v1.txt


jerryfat commented 7 months ago

I put on github for now: https://raw.githubusercontent.com/jerryfat/FIX-Gateway/master/list-modules-imported.py https://raw.githubusercontent.com/jerryfat/FIX-Gateway/master/list-pip-module-deps.py

$ pythonlist-pip-module-deps.py -m module arg lists module deps

$ python list-modules-imported.py if no args creates 'sysroot.toml style 'component text of itself and if any dependency packages also formatted as components for sysroot.toml with args $ list-modules-imported.py -p package or $ list-modules-imported.py -f filename.py , either creates sysroot.toml style [component] text

-Jerry Fat

 On Friday, February 23, 2024 at 10:48:26 PM EST, Achille MARTIN ***@***.***> wrote:  

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

achille-martin commented 7 months ago

Impressive, thanks for figuring this out!

Glad we are now able to import non-standard python packages through wheels.

I'll create a merge request so we can fully test the feature and then merge when happy.

In the meantime, please make use of your script to get the python packages you need!

Thanks again

jerryfat commented 7 months ago

Hi Achille: I just fixed a bug after testing list-modules-imported,py ; error when if wheels dir not existing also tried all the cmd line args, -f -m none and with wheels dir existing you;ll need to pull new copy of list-modules-imported,py https://raw.githubusercontent.com/jerryfat/FIX-Gateway/master/list-modules-imported.py if you find issues let me know Thanks and Finest Regards, -Jerry Fat

jerryfat commented 7 months ago

Hi: Do you know if is there is a set of instruction like yours but using pyqtdeploy2.5.1 and .pdy for building an apk ? The pyqtdeply2.5.1 looked great and it generates a huge json file that looks like it has tons of things going on inside. any idea when did the change occur from .pdy to .pdt config files ? Thanks in advance, -Jerry Fat

achille-martin commented 7 months ago

There is unfortunately nothing to my knowledge that deals with .apk building and pyqtdeploy-2.5.1.

This is actually one of the reasons why I started PyQt-CroM: so that we can build on any platform quickly and easily.

I am not sure about the change from .pdy to .pdt, only the great Phil Thompson will know... I haven't investigated the code fully yet, but I hope to get more insights as I use pyqtdeploy tool.

In the meantime, if you are looking for more documentation on pyqtdeploy-2.5.1, I suggest that you:

Additional note

Thinking about the documentation, I have double checked the ChangeLog for pyqtdeploy-3.3.0 (located in pyqtdeploy-3.3.0/ChangeLog) and it looks like there has been a full switch from .pdy to .pdt for all related files (even test files). This could mean that the upgrade from pyqtdeploy project file .pdy to pyqtdeploy project file .pdt might have been for consistency across the package.

jerryfat commented 7 months ago

Hi Achille: Does this sound correct ?

to test if the pyqtdeploy3.3.0 docs and instructions work for adding 'component' certifi in sysroot.toml

I added 'import certifi' into your demo_app.py downloaded .wh built binariesl works in linux nbut ot in android and crashes

summary, from ~/Documents/pyqt-crom/examples/demo/demo_project/ sysroot.toml file:

[certifi] plugin = "wheel" wheel = "certifi-2021.10.8-py2.py3-none-any.whl" dependencies = ["Python:importlib.resources", "Python:os"] exclusions = ["main.py"]

I tried to test adding componet by importing into demo app: I added "import certifi" to demo app.py and downloaded the .whl

$ cd ~/Documents/pyqt-crom/examples/demo/demo_project/ $ python -m pip download --only-binary :all: --dest . --no-cache certifi to check .whl: $ ls ~/Documents/pyqt-crom/examples/demo/demo_project/certifi-2024.2.2-py3-none-any.whl -als $ 160 -rw-rw-r-- 1 jf jf 163774 Feb 28 17:44 /home/jf/Documents/pyqt-crom/examples/demo/demo_project/certifi-2024.2.2-py3-none-any.whl

using your instructions pyqtdeploy3.3.0 build succeeds , demo_app.py works in linux with the import certifi demo_app . apk crashes in android (linux android studio emulator and my phone) demo_app.py witout the import certifi wokrks on android and linux

I dont know how to move forward now. btw in that app list-modules-imported.py i wrote that gets .whl files on github , it still needs dependencies added from the list-pip-module-deps.py test qapp I wrote on github I was going to combine but,

If the import certifi in demo_app.py fails then I dont know what to do now.

I tried to build using the .pdy and pqyqtdeply2.5.1 and it failed to build sysroot and errored

Do you have a way to contact author ? I tried to get a qt forum login but no one ever responded, and login doesnt work Thanks -Jerry Fat

achille-martin commented 7 months ago

This is work in progress, I'll have a look but I can't say when it will be fully fixed.

I can do some testing before end of March, but it will be tight to get the feature completely ready.

The best way to get ultra quick feedback is to send an email to Riverbank / the PyQt maillist: https://www.riverbankcomputing.com/mailman/listinfo/pyqt

I think you should be able to send an email directly to pyqt@riverbankcomputing.com.

achille-martin commented 6 months ago

Hi @jerryfat,

Intro

I have created a small example on how to integrate PyYAML into an Android app.

The example pyqt5 app can be found on branch 32-bug-cannot-add-python-packages-from-site-packages-directory-of-my-python-interpreter-to-the-pdt-of-pyqtdeploy-330.

Question

Can you try to generate an Android app from the pyqt5 script and tell me if it all works as you expect?

Notes

You might need to:

cd $PYQT_CROM
git pull
git checkout 32-bug-cannot-add-python-packages-from-site-packages-directory-of-my-python-interpreter-to-the-pdt-of-pyqtdeploy-330
source $PYQT_CROM_DIR/venv/pyqt-crom-venv/bin/activate
pip3 install -r requirements.txt
cd $PYQT_CROM_DIR/utils
&& python3 build_app.py --pdt $PYQT_CROM_DIR/examples/external/external_python_project/config.pdt --jobs 1 --target android-64 --qmake $QT_DIR/android/bin/qmake --verbose

Going further

If you confirm that it's all going as expected, I will add a little tutorial on how to add any external python package to the sysroot.toml and config.pdt.

jerryfat commented 6 months ago

HI: Thanks for the info on building android app with python package, I will try and get back to you friday THANKS

mrRaduga commented 5 months ago

Hello @achille-martin

I am trying to use your tool now and I concluded that I also need the possibility to add extra packages, like requests, numpy, pandas. So, this why now I am here.

I have already tested your proposal and first I got the following error in the build step:

Screenshot 2024-04-18 145658

And I realized that the correct yaml package name is PyYaml and changed it to sysroot.toml and config.pdt. After that the .apk file builds successfully but unfortunately it doesn't run on Android.

achille-martin commented 5 months ago

Hi @mrRaduga,

I have created a pull request that demonstrates the capability you are looking for, for the yaml case.

I am in the process of adding a bit of documentation to the PR so that anyone can add any non-standard python module to a PyQt5 app.

I'll keep adding some documentation and will ask you later on to verify that you can access the capability you need.

I expect that I will be done by the end of the month (April 2024).

achille-martin commented 5 months ago

After re-reading your message, I realised that you have started testing the PR.

I will review the issue with PyYAML. This is in fact an exception of the whole process.

The name on PyPI (PyYAML) does not match the name of the package locally installed (yaml).

To make sure that we are on the same page: 1) Can you tell me what you have done? Like all the steps

2) Have you used this sysroot and this pdt to build this example app with yaml?

jerryfat commented 5 months ago

Hello Achille: I was able to compile after changing package name too the app compiled, but the app and apk did not work on Android or simulator in linux. Was this techinque ever tested ? It doesnt work here too. IMHO This is not a useabale solution for running python packages on Android. my $0.02 -Jerry Fat

 On Thursday, April 18, 2024 at 11:54:44 AM EDT, Achille MARTIN ***@***.***> wrote:  

After re-reading your message, I realised that you have started testing the PR.

I will review the issue with PyYAML. This is in fact an exception of the whole process.

The name on PyPI does not match the name of the package locally installed.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

achille-martin commented 5 months ago

Changing the name of the dependency in the sysroot only ignores the module. This is because the build process will look for something called PyYAML, but the only module that is installed locally is yaml. If you look closely at the debug prints of pyqtdeploy, you will realise that yaml is never built or never included at any point. It is simply skipped.

I might have missed an instruction or step somewhere, because I can definitely create an Android app with yaml.

I have attached the Android app example, which corresponds to this PyQt5 app. You can run the app in an Emulator or on an Android phone to validate.

ExternalPythonApp.zip

However, that is not the interesting bit. The interesting bit is that other users cannot replicate the steps, so I need to figure out where it went wrong.

mrRaduga commented 5 months ago

For testing external packages feature I did the following steps:

  1. Download your example external/external_python_project from the second branch
  2. Install the PyYAML library by the command pip install PyYAML in the enviroment
  3. Tried to deploy to android by command python3 build_app.py --pdt $HOME/Documents/pyqt-crom/examples/external/external_python_project/config.pdt --jobs 1 --target android-32 --qmake $QT_DIR/android/bin/qmake --verbose
  4. Received the error showed in my last message
  5. Updated the name of library in sysroot.toml file, as bellow: image
  6. Update the name of library in config.pdt file, as bellow: Screenshot 2024-04-19 100452
  7. Tried again to deploy to Android, which builds successfully, .apk file installed successfully but didn't run.

You are right, I checked the logs of building process and yaml is never included. And what is interesting the pip list command shows me the yaml module with PyYAML name, but the files locally are installed in yaml folder, no clue why. Maybe take sense to try with another modules?

Based on that I tried again with your config.pdt and sysroot.toml files and see in the console the rows bellow:

image

But I didn't see the package yaml in the build-android or sysroot-android folder after building process.

achille-martin commented 5 months ago

Hi @jerryfat and @mrRaduga,

I am still in the process of trying to figure out what is preventing you from building with yaml on your machines.

I have created a help script that collects all necessary information for debugging purposes.

Can you please run the following commands and paste / link the output on this issue thread?

1) Get wheel name to validate platform and version

chmod +x $PYQT_CROM_DIR/utils/bash/get_python_package_wheel.sh &&
$PYQT_CROM_DIR/utils/bash/get_python_package_wheel.sh pyyaml 6.0.1

Note: the output will be on screen. Please copy-paste the output here.

2) Get errors linked to building and background information about OS and repo (change <absolute_path_to_your_pdt>)

chmod +x $PYQT_CROM_DIR/utils/bash/get_command_output_help.sh &&
$PYQT_CROM_DIR/utils/bash/get_command_output_help.sh "cd $PYQT_CROM_DIR/utils/python && python3 build_app.py --pdt <absolute_path_to_your_pdt>/config.pdt --jobs 1 --target android-64 --qmake $QT_DIR/android/bin/qmake --verbose"

If you would like to add your sysroot.toml and config.pdt to the output of the script (so that you don't have to copy-paste them here):

chmod +x $PYQT_CROM_DIR/utils/bash/get_command_output_help.sh &&
$PYQT_CROM_DIR/utils/bash/get_command_output_help.sh "cd $PYQT_CROM_DIR/utils/python && python3 build_app.py --pdt <absolute_path_to_your_pdt>/config.pdt --jobs 1 --target android-64 --qmake $QT_DIR/android/bin/qmake --verbose" -f "<absolute_path_to_sysroot>/sysroot.toml <absolute_path_to_pdt>/config.pdt"

WARNING: there are 3 breaking changes now in the PR (that will be released as v3.0.0).

mrRaduga commented 5 months ago

Good morning @achille-martin,

I have successfully run your commands, and you can check the output in the attached file. For testing I have used the external_package_project_2 without any changes for config.pdt and sysroot.toml file. The .apk file successfully installed and launched on Android device with API 31, see the screenshot below and attached log from the device. Screenshot_20240423_100920

command_output_help_2024_04_23-09_59_41.md

pyqt5_app_with_yaml.log

Physically located in this path on the device: /data/data/org.qtproject.example.ExternalPyApp/files/pyqt5_app_data/pyqt5_app_with_yaml/pyqt5_app_with_yaml.log

achille-martin commented 5 months ago

Thank you @mrRaduga, that is very promising!

I will keep updating the README and add more instructions to let people know how to add their own non-standard python modules.

I might give it a try with pandas in the meantime because it has various python package dependencies (while PyYAML had none).

I will also sort out this issue, probably once this issue is merged.

This will overall take a bit of time and I am away for a few weeks (so I would estimate the updates to be done by mid May 2024).

mrRaduga commented 4 months ago

Servus @achille-martin

Nice to find you well and thanks for your job

Based on your updates, I tried again to implement the request package in the external example, but unfortunately without any good result.

Getting the wheels for a request package using the get_python_package_wheel.sh script returns me the following result, which should not be underestimated. Why do I have 5 wheels founded and downloaded, are these dependencies for the requests package? If yes, why then are there no others? image

After that I tried to get the dependencies for each package using the py_package_dependent_collector.py script. Finally, the sysroot file looks like this, which is a little off-putting in my opinion:

image

As I understand it, from this point of view, each dependency that is a non-standard Python library also requires specifying all the wheels and dependencies in the sysroot file. If I understand this correctly, this can be very tedious, recursive, and hard to debug which packages are missing as a dependency.

P.S. I also tried to just specify the requests package as a wheel in the sysroot file, also without success.

I'll be glad if you come back with a successful example of using the requests library, I see it as a rather complex example.

achille-martin commented 4 months ago

@mrRaduga Thanks for giving it a try.

I am currently testing with numpy (no required dependencies but complex) and then pandas (required dependencies and complex so should be equivalent to requests). I am also in the process of creating a script that will search for all the dependencies and list them so they can be added to the sysroot directly.

So far, I am having a few issues with some python built-in packages (potential need to move to Python3.11.0). I'll update you when I can get the numpy package working and I'll share the handy scripts.

jerryfat commented 4 months ago

Hello Achille: I also am unable to include packages in the .pdt file I have also written two python apps that would get package and download them, but they become recursive after getting all the packages and deps in python literally, and besides includiing them in sysroot never worked either there is an already existing package (excilim) and it does not get built into android binary either that should be first test try date package and print the time or date in android using the example, it wont work either in android as package never gets built and then linked into android binary that should be the first test -Jerry Fat

list-modules-imported-for-pdt.py

Cf. file list-modules-imported-for-pdt-v1.txt

list-modules-imported-for-pdt-v1.txt

achille-martin commented 4 months ago

@jerryfat Thank you for your contribution.

The wheels do not need to be downloaded actually, pyqtdeploy will take care of that and build straight away.

I am currently trying to understand whether specific standard python library dependencies need to be specified for a non-standard python package or if it would be better to include ALL standard python libraries into the app (at the cost of the heavy weight of the app).

To remind everyone, pyqt-crom is still in its early stages, so there is a lot of work required around understand the "build process" (especially through Qt tools).