kivy / python-for-android

Turn your Python application into an Android APK
https://python-for-android.readthedocs.io
MIT License
8.35k stars 1.85k forks source link

`charset-normalizer >= 3.0.0` is not plain-python anymore, and requires a recipe #2755

Open snowgato opened 1 year ago

snowgato commented 1 year ago

Checklist

Versions

Description

The app should display repeatedly some stock quotes using Kivy and yahooquery module. It works on windows. Buildozer build process produces an apk. When launched on the smartphone, the app window doesn't show up. adb logs report the following error:

02-21 18:02:40.903 12625 12674 I python : ImportError: dlopen failed: "/data/data/ced.bart.bourse/files/app/_python_bundle/site-packages/charset_normalizer/md.so" is for EM_X86_64 (62) instead of EM_AARCH64 (183)
02-21 18:02:40.903 12625 12674 I python : Python for android ended.

buildozer.spec

Command:

buildozer -v android debug

Spec file:

[app]
title = Bourse
package.name = Bourse
package.domain = ced.bart
source.dir = .
source.include_exts = py,png,jpg,kv,atlas,json,wav
version = 0.1
requirements=python3,kivy==2.1.0,datetime,yahooquery,pandas,lxml,numpy,python-dateutil,six,pytz,requests-futures,requests,certifi,charset-normalizer,idna,urllib3,tqdm
orientation = portrait
osx.python_version = 3
osx.kivy_version = 2.1.0
fullscreen = 0
android.archs = arm64-v8a, armeabi-v7a
android.allow_backup = True
ios.kivy_ios_url = https://github.com/kivy/kivy-ios
ios.kivy_ios_branch = master
ios.ios_deploy_url = https://github.com/phonegap/ios-deploy
ios.ios_deploy_branch = 1.10.0
ios.codesign.allowed = false
[buildozer]
log_level = 2
warn_on_root = 1

Logs

Tue Feb 21 19:02:14 CET 2023
# Check configuration tokens
# Ensure build layout
# Check configuration tokens
# Preparing build
# Check requirements for android
# Search for Git (git)
#  -> found at /usr/bin/git
# Search for Cython (cython)
#  -> found at /home/ced/.local/bin/cython
# Search for Java compiler (javac)
#  -> found at /usr/lib/jvm/java-17-openjdk-amd64/bin/javac
# Search for Java keytool (keytool)
#  -> found at /usr/lib/jvm/java-17-openjdk-amd64/bin/keytool
# Install platform
# Run ['git', 'config', '--get', 'remote.origin.url']
# Cwd /home/ced/programmation/python/kivy/kivy_bourse/.buildozer/android/platform/python-for-android
https://github.com/kivy/python-for-android.git
# Run ['git', 'branch', '-vv']
# Cwd /home/ced/programmation/python/kivy/kivy_bourse/.buildozer/android/platform/python-for-android
* master 56bf7335 [origin/master] Merge pull request #2747 from misl6/release-2023.02.10
# Run ['/usr/bin/python', '-m', 'pip', 'install', '-q', '--user', 'appdirs', 'colorama>=0.3.3', 'jinja2', 'sh>=1.10, <2.0; sys_platform!="nt"', 'pep517', 'toml', 'packaging']
# Cwd None
WARNING: Ignoring invalid distribution -ip (/home/ced/.local/lib/python3.8/site-packages)
WARNING: Ignoring invalid distribution -ip (/home/ced/.local/lib/python3.8/site-packages)
WARNING: Ignoring invalid distribution -ip (/home/ced/.local/lib/python3.8/site-packages)
WARNING: Ignoring invalid distribution -ip (/home/ced/.local/lib/python3.8/site-packages)
WARNING: Ignoring invalid distribution -ip (/home/ced/.local/lib/python3.8/site-packages)
WARNING: Ignoring invalid distribution -ip (/home/ced/.local/lib/python3.8/site-packages)

[notice] A new release of pip available: 22.3.1 -> 23.0
[notice] To update, run: python -m pip install --upgrade pip
# Apache ANT found at /home/ced/.buildozer/android/platform/apache-ant-1.9.4
# Android SDK found at /home/ced/.buildozer/android/platform/android-sdk
# Recommended android's NDK version by p4a is: 25b
# Android NDK found at /home/ced/.buildozer/android/platform/android-ndk-r25b
# Run ['/usr/bin/python', '-m', 'pythonforandroid.toolchain', 'aab', '-h', '--color=always', '--storage-dir=/home/ced/programmation/python/kivy/kivy_bourse/.buildozer/android/platform/build-arm64-v8a_armeabi-v7a', '--ndk-api=21', '--ignore-setup-py', '--debug']
# Cwd /home/ced/programmation/python/kivy/kivy_bourse/.buildozer/android/platform/python-for-android
[WARNING]: prerequisites.py is experimental and does not support all prerequisites yet.
[WARNING]: Please report any issues to the python-for-android issue tracker.
/home/ced/programmation/python/kivy/kivy_bourse/.buildozer/android/platform/python-for-android/pythonforandroid/toolchain.py:59: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if LooseVersion(cur_ver) < LooseVersion(version):
[WARNING]: prerequisites.py is experimental and does not support all prerequisites yet.
[WARNING]: Please report any issues to the python-for-android issue tracker.
usage: toolchain.py aab [-h] [--debug] [--color {always,never,auto}]
                        [--sdk-dir SDK_DIR] [--ndk-dir NDK_DIR]
                        [--android-api ANDROID_API]
                        [--ndk-version NDK_VERSION] [--ndk-api NDK_API]
                        [--symlink-bootstrap-files]
                        [--storage-dir STORAGE_DIR] [--arch ARCH]
                        [--dist-name DIST_NAME] [--requirements REQUIREMENTS]
                        [--recipe-blacklist RECIPE_BLACKLIST]
                        [--blacklist-requirements BLACKLIST_REQUIREMENTS]
                        [--bootstrap BOOTSTRAP] [--hook HOOK] [--force-build]
                        [--no-force-build] [--require-perfect-match]
                        [--no-require-perfect-match] [--allow-replace-dist]
                        [--no-allow-replace-dist]
                        [--local-recipes LOCAL_RECIPES]
                        [--activity-class-name ACTIVITY_CLASS_NAME]
                        [--service-class-name SERVICE_CLASS_NAME]
                        [--java-build-tool {auto,ant,gradle}] [--copy-libs]
                        [--no-copy-libs] [--add-asset ASSETS]
                        [--add-resource RESOURCES] [--private PRIVATE]
                        [--use-setup-py] [--ignore-setup-py] [--release]
                        [--with-debug-symbols] [--keystore KEYSTORE]
                        [--signkey SIGNKEY] [--keystorepw KEYSTOREPW]
                        [--signkeypw SIGNKEYPW]

optional arguments:
  -h, --help            show this help message and exit
  --debug               Display debug output and all build info
  --color {always,never,auto}
                        Enable or disable color output (default enabled on
                        tty)
  --sdk-dir SDK_DIR, --sdk_dir SDK_DIR
                        The filepath where the Android SDK is installed
  --ndk-dir NDK_DIR, --ndk_dir NDK_DIR
                        The filepath where the Android NDK is installed
  --android-api ANDROID_API, --android_api ANDROID_API
                        The Android API level to build against defaults to 33
                        if not specified.
  --ndk-version NDK_VERSION, --ndk_version NDK_VERSION
                        DEPRECATED: the NDK version is now found automatically
                        or not at all.
  --ndk-api NDK_API     The Android API level to compile against. This should
                        be your *minimal supported* API, not normally the same
                        as your --android-api. Defaults to min(ANDROID_API,
                        21) if not specified.
  --symlink-bootstrap-files, --ssymlink_bootstrap_files
                        If True, symlinks the bootstrap files creation. This
                        is useful for development only, it could also cause
                        weird problems.
  --storage-dir STORAGE_DIR
                        Primary storage directory for downloads and builds
                        (default: /home/ced/.local/share/python-for-android)
  --arch ARCH           The archs to build for.
  --dist-name DIST_NAME, --dist_name DIST_NAME
                        The name of the distribution to use or create
  --requirements REQUIREMENTS
                        Dependencies of your app, should be recipe names or
                        Python modules. NOT NECESSARY if you are using Python
                        3 with --use-setup-py
  --recipe-blacklist RECIPE_BLACKLIST
                        Blacklist an internal recipe from use. Allows
                        disabling Python 3 core modules to save size
  --blacklist-requirements BLACKLIST_REQUIREMENTS
                        Blacklist an internal recipe from use. Allows
                        disabling Python 3 core modules to save size
  --bootstrap BOOTSTRAP
                        The bootstrap to build with. Leave unset to choose
                        automatically.
  --hook HOOK           Filename to a module that contains python-for-android
                        hooks
  --local-recipes LOCAL_RECIPES, --local_recipes LOCAL_RECIPES
                        Directory to look for local recipes
  --activity-class-name ACTIVITY_CLASS_NAME
                        The full java class name of the main activity
  --service-class-name SERVICE_CLASS_NAME
                        Full java package name of the PythonService class
  --java-build-tool {auto,ant,gradle}
                        The java build tool to use when packaging the APK,
                        defaults to automatically selecting an appropriate
                        tool.
  --add-asset ASSETS    Put this in the assets folder in the apk.
  --add-resource RESOURCES
                        Put this in the res folder in the apk.
  --private PRIVATE     the directory with the app source code files
                        (containing your main.py entrypoint)
  --use-setup-py        Process the setup.py of a project if present.
                        (Experimental!
  --ignore-setup-py     Don't run the setup.py of a project if present. This
                        may be required if the setup.py is not designed to
                        work inside p4a (e.g. by installing dependencies that
                        won't work or aren't desired on Android
  --release             Build your app as a non-debug release build. (Disables
                        gdb debugging among other things)
  --with-debug-symbols  Will keep debug symbols from `.so` files.
  --keystore KEYSTORE   Keystore for JAR signing key, will use jarsigner
                        default if not specified (release build only)
  --signkey SIGNKEY     Key alias to sign PARSER_APK. with (release build
                        only)
  --keystorepw KEYSTOREPW
                        Password for keystore
  --signkeypw SIGNKEYPW
                        Password for key alias

  Whether to force compilation of a new distribution

  --force-build
  --no-force-build      (this is the default)
  --require-perfect-match
  --no-require-perfect-match
                        (this is the default)
  --allow-replace-dist  (this is the default)
  --no-allow-replace-dist
  --copy-libs
  --no-copy-libs        (this is the default)
# Check application requirements
# Compile platform
# Run ['/usr/bin/python', '-m', 'pythonforandroid.toolchain', 'create', '--dist_name=Bourse', '--bootstrap=sdl2', '--requirements=python3,kivy==2.1.0,datetime,yahooquery,pandas,lxml,numpy,python-dateutil,six,pytz,requests-futures,requests,certifi,charset-normalizer==2.1.1,idna,urllib3,tqdm', '--arch=arm64-v8a', '--arch=armeabi-v7a', '--copy-libs', '--color=always', '--storage-dir=/home/ced/programmation/python/kivy/kivy_bourse/.buildozer/android/platform/build-arm64-v8a_armeabi-v7a', '--ndk-api=21', '--ignore-setup-py', '--debug']
# Cwd /home/ced/programmation/python/kivy/kivy_bourse/.buildozer/android/platform/python-for-android
[WARNING]: prerequisites.py is experimental and does not support all prerequisites yet.
[WARNING]: Please report any issues to the python-for-android issue tracker.
/home/ced/programmation/python/kivy/kivy_bourse/.buildozer/android/platform/python-for-android/pythonforandroid/toolchain.py:59: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if LooseVersion(cur_ver) < LooseVersion(version):
[WARNING]: prerequisites.py is experimental and does not support all prerequisites yet.
[WARNING]: Please report any issues to the python-for-android issue tracker.
[INFO]:    Recipe kivy: version "2.1.0" requested
[INFO]:    Recipe charset-normalizer: version "2.1.1" requested
[WARNING]: $ANDROIDNDKVER is deprecated and no longer necessary, the value you set is ignored
[INFO]:    Will compile for the following archs: armeabi-v7a, arm64-v8a
[INFO]:    Found Android API target in $ANDROIDAPI: 31
[INFO]:    Available Android APIs are (31)
[INFO]:    Requested API target 31 is available, continuing.
[INFO]:    Found NDK dir in $ANDROIDNDK: /home/ced/.buildozer/android/platform/android-ndk-r25b
[INFO]:    Found NDK version 25b
[INFO]:    Getting NDK API version (i.e. minimum supported API) from user argument
[INFO]:    Of the existing distributions, the following meet the given requirements:
[INFO]:         Bourse: min API 21, includes recipes (hostpython3, libbz2, libffi, liblzma, librt, libxml2, openssl, sdl2_image, sdl2_mixer, sdl2_ttf, sqlite3, libxslt, python3, sdl2, pytz, setuptools, cython, lxml, six, numpy, pyjnius, android, kivy, pandas, tqdm, requests, urllib3, datetime, charset-normalizer, certifi, requests-futures, yahooquery, python-dateutil, idna), built for archs (armeabi-v7a, arm64-v8a)
[INFO]:    Bourse has compatible recipes, using this one
# Build the application #8
# Copy application source from /home/ced/programmation/python/kivy/kivy_bourse
# Create directory /home/ced/programmation/python/kivy/kivy_bourse/.buildozer/android/app
# Copy /home/ced/programmation/python/kivy/kivy_bourse/main.py
# Copy /home/ced/programmation/python/kivy/kivy_bourse/recurse.py
# Package the application
# project.properties updated
# Run ['/usr/bin/python', '-m', 'pythonforandroid.toolchain', 'apk', '--bootstrap', 'sdl2', '--dist_name', 'Bourse', '--name', 'Bourse', '--version', '0.1', '--package', 'ced.bart.bourse', '--minsdk', '21', '--ndk-api', '21', '--private', '/home/ced/programmation/python/kivy/kivy_bourse/.buildozer/android/app', '--android-entrypoint', 'org.kivy.android.PythonActivity', '--android-apptheme', '@android:style/Theme.NoTitleBar', '--orientation', 'portrait', '--window', '--enable-androidx', '--copy-libs', '--arch', 'arm64-v8a', '--arch', 'armeabi-v7a', '--color=always', '--storage-dir=/home/ced/programmation/python/kivy/kivy_bourse/.buildozer/android/platform/build-arm64-v8a_armeabi-v7a', '--ndk-api=21', '--ignore-setup-py', '--debug']
# Cwd /home/ced/programmation/python/kivy/kivy_bourse/.buildozer/android/platform/python-for-android
[WARNING]: prerequisites.py is experimental and does not support all prerequisites yet.
[WARNING]: Please report any issues to the python-for-android issue tracker.
/home/ced/programmation/python/kivy/kivy_bourse/.buildozer/android/platform/python-for-android/pythonforandroid/toolchain.py:59: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if LooseVersion(cur_ver) < LooseVersion(version):
[WARNING]: prerequisites.py is experimental and does not support all prerequisites yet.
[WARNING]: Please report any issues to the python-for-android issue tracker.
[WARNING]: $ANDROIDNDKVER is deprecated and no longer necessary, the value you set is ignored
[INFO]:    Will compile for the following archs: armeabi-v7a, arm64-v8a
[INFO]:    Found Android API target in $ANDROIDAPI: 31
[INFO]:    Available Android APIs are (31)
[INFO]:    Requested API target 31 is available, continuing.
[INFO]:    Found NDK dir in $ANDROIDNDK: /home/ced/.buildozer/android/platform/android-ndk-r25b
[INFO]:    Found NDK version 25b
[INFO]:    Getting NDK API version (i.e. minimum supported API) from user argument
[INFO]:    Of the existing distributions, the following meet the given requirements:
[INFO]:         Bourse: min API 21, includes recipes (hostpython3, libbz2, libffi, liblzma, librt, libxml2, openssl, sdl2_image, sdl2_mixer, sdl2_ttf, sqlite3, libxslt, python3, sdl2, pytz, setuptools, cython, lxml, six, numpy, pyjnius, android, kivy, pandas, tqdm, requests, urllib3, datetime, charset-normalizer, certifi, requests-futures, yahooquery, python-dateutil, idna), built for archs (armeabi-v7a, arm64-v8a)
[INFO]:    Bourse has compatible recipes, using this one
[INFO]:    Of the existing distributions, the following meet the given requirements:
[INFO]:         Bourse: min API 21, includes recipes (hostpython3, libbz2, libffi, liblzma, librt, libxml2, openssl, sdl2_image, sdl2_mixer, sdl2_ttf, sqlite3, libxslt, python3, sdl2, pytz, setuptools, cython, lxml, six, numpy, pyjnius, android, kivy, pandas, tqdm, requests, urllib3, datetime, charset-normalizer, certifi, requests-futures, yahooquery, python-dateutil, idna), built for archs (armeabi-v7a, arm64-v8a)
[INFO]:    Bourse has compatible recipes, using this one
[INFO]:    -> directory context /home/ced/programmation/python/kivy/kivy_bourse/.buildozer/android/platform/build-arm64-v8a_armeabi-v7a/build/bootstrap_builds/sdl2
[INFO]:    <- directory context /home/ced/programmation/python/kivy/kivy_bourse/.buildozer/android/platform/python-for-android
[INFO]:    -> directory context /home/ced/programmation/python/kivy/kivy_bourse/.buildozer/android/platform/build-arm64-v8a_armeabi-v7a/dists/Bourse
Compiling '/home/ced/programmation/python/kivy/kivy_bourse/.buildozer/android/app/main.py'...
Compiling '/home/ced/programmation/python/kivy/kivy_bourse/.buildozer/android/app/sitecustomize.py'...
Compiling '/home/ced/programmation/python/kivy/kivy_bourse/.buildozer/android/app/recurse.py'...
[INFO]:    Detected highest available build tools version to be 34.0.0-rc1
[DEBUG]:   -> running dos2unix /home/ced/programmation/python/kivy/kivy_bourse/.buildozer/android/platform/build-arm64-v8a_armeabi-v7a/dists/Bourse/gradlew
[DEBUG]:        dos2unix: converting file /home/ced/programmation/python/kivy/kivy_bourse/.buildozer/android/platform/build-arm64-v8a_armeabi-v7a/dists/Bourse/gradlew to Unix format...
[DEBUG]:   -> running gradlew clean assembleDebug
[DEBUG]:
[DEBUG]:        > Configure project :
[DEBUG]:        WARNING:Using flatDir should be avoided because it doesn't support any meta-data formats.
[DEBUG]:
[DEBUG]:        > Task :compileDebugJavaWithJavac
[DEBUG]:        Note: Some input files use or override a deprecated API.
[DEBUG]:        Note: Recompile with -Xlint:deprecation for details.
[DEBUG]:
[DEBUG]:        > Task :stripDebugDebugSymbols
[DEBUG]:        Unable to strip the following libraries, packaging them as they are: libSDL2.so, libSDL2_image.so, libSDL2_mixer.so, libSDL2_ttf.so, libbz2.so, libc++_shared.so, libcrypto1.1.so, libffi.so, liblzma.so, libmain.so, libpybundle.so, libpython3.9.so, libsqlite3.so, libssl1.1.so.
[DEBUG]:
[DEBUG]:        Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
[DEBUG]:
[DEBUG]:        You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.DLE
[DEBUG]:
[DEBUG]:        See https://docs.gradle.org/7.4.1/userguide/command_line_interface.html#sec:command_line_warnings
[DEBUG]:
[DEBUG]:        BUILD SUCCESSFUL in 18s
[DEBUG]:        31 actionable tasks: 31 executed

[INFO]:    <- directory context /home/ced/programmation/python/kivy/kivy_bourse/.buildozer/android/platform/python-for-android
[INFO]:    Of the existing distributions, the following meet the given requirements:
[INFO]:         Bourse: min API 21, includes recipes (hostpython3, libbz2, libffi, liblzma, librt, libxml2, openssl, sdl2_image, sdl2_mixer, sdl2_ttf, sqlite3, libxslt, python3, sdl2, pytz, setuptools, cython, lxml, six, numpy, pyjnius, android, kivy, pandas, tqdm, requests, urllib3, datetime, charset-normalizer, certifi, requests-futures, yahooquery, python-dateutil, idna), built for archs (armeabi-v7a, arm64-v8a)
[INFO]:    Bourse has compatible recipes, using this one
[INFO]:    # Copying android package to current directory
[INFO]:    # Android package filename not found in build output. Guessing...
[INFO]:    # Found android package file: /home/ced/programmation/python/kivy/kivy_bourse/.buildozer/android/platform/build-arm64-v8a_armeabi-v7a/dists/Bourse/build/outputs/apk/debug/Bourse-debug.apk
[INFO]:    # Add version number to android package
[INFO]:    # Android package renamed to Bourse-debug-0.1.apk
[DEBUG]:   -> running cp /home/ced/programmation/python/kivy/kivy_bourse/.buildozer/android/platform/build-arm64-v8a_armeabi-v7a/dists/Bourse/build/outputs/apk/debug/Bourse-debug.apk Bourse-debug-0.1.apk
WARNING: Received a --sdk argument, but this argument is deprecated and does nothing.
No setup.py/pyproject.toml used, copying full private data into .apk.
Applying Java source code patches...
Applying patch: src/patches/SDLActivity.java.patch
Warning: failed to apply patch (exit code 1), assuming it is already applied:  src/patches/SDLActivity.java.patch
# Android packaging done!
# APK Bourse-0.1-arm64-v8a_armeabi-v7a-debug.apk available in the bin directory
misl6 commented 1 year ago

Hi @snowgato !

charset-normalizer >= 3.0.0 is not plain python (anymore, it was previously), so it will require a recipe to build.

As a workaround, you can use a different version, or chardet, which is plain-python.

misl6 commented 1 year ago

P.S. : Changed the title, and added the feature-request label.

snowgato commented 1 year ago

Ok, meanwhile someone gave me the advice to use charset_normalizer=2.1.1 , solving this bug. Thanks ! By the way, I've had previous error messages stating that some libraries were missing. Those libraries were libraries required by one of the modules that I imported, they were not libraries that I had directly imported in the main.py file. As I was fed up of using adb logcat -s python to see missing libraries making crash the app, one after the other from the different recursively imported libraries from yahooquery module. I've finally made a small program to recursively list all the imported libraries from yahooquery module. Should'nt buildozer be able to cope with these libraries once put in requirements field of buildozer.spec only the libraries directly imported in the main.py ? Or do we have to list extensively all the imported libraries directly in main.py but also from the libraries imported in main.py ? Thanks !

TonyWeston commented 1 year ago

If I am creating an AWS Lambda layer, which includes charset-normalizer-3.2.0 installed with 'pip install', then the hashcode of this layer will change every time it is built.

Which means it is impossible to automatically deploy a lambda layer only when it changes.

This is a big deal. Given a set of unchanging input source codes, the target should not change. I believe its due to bin/normalizer.exe being compiled each time, and the build date being stored in there.

Not sure what the workaround for this should be... but its really crappy.