search

flet-dev / serious-python

Python runtime for Flutter apps
Apache License 2.0
212 stars 22 forks source link

How Far Can I Go with Machine Learning? #25

Closed certainlyWrong closed 1 year ago

certainlyWrong commented 1 year ago

I tested some audio feature extraction libraries, such as Librosa. When running the MFCC function, the application immediately crashed. I also attempted to run a heavier library like tensorflow-cpu and train a simple embedded model, but the application crashed once again. Of course, we're dealing with different architectures here, and I believe it might not be straightforward.

In any case, I would like to understand the extent to which we intend to address this issue. What are your thoughts on what we can achieve? I believe that at the very least, Librosa's MFCC should run smoothly, but beneath the surface, it relies on dynamic libraries and not solely on pure Python code.

FeodorFitsner commented 1 year ago

serious_python depends on Kivy, so a native library you'd like to use must have a "recipe" here: https://github.com/kivy/python-for-android/tree/develop/pythonforandroid/recipes. Submit a new issue at https://github.com/kivy/python-for-android/issues.

certainlyWrong commented 1 year ago

Okay, this is enough to address my doubts. I will read the content you recommended. Thank you <3.

certainlyWrong commented 1 year ago

Screenshot_20230926_123416

According to Python-for-Android, it's possible to run numpy, but that doesn't happen in Serious-Python. Do you have any idea why this happens?

mohammedhashim790 commented 1 year ago
serious_python_app

Importing Python using the editor, I get the following error. It seems Numpy is not able find the C-extensions required. The following command was used to package the python app. The import fails on IOS and Android both.

dart run serious_python:main package app/src

FeodorFitsner commented 1 year ago

Because Numpy is a non-pure Python module, i.e. part of it is written in C/Rust. Non-pure modules should be built as part of Kivy.

I'm going to provide all necessary instructions how to do that.

mohammedhashim790 commented 1 year ago

Because Numpy is a non-pure Python module, i.e. part of it is written in C/Rust. Non-pure modules should be built as part of Kivy.

I'm going to provide all necessary instructions how to do that.

Hi!. Thanks for the reply. But it would be nice, if it is shared at the earliest.

FeodorFitsner commented 1 year ago

https://github.com/flet-dev/serious-python/tree/main/src/serious_python#adding-custom-python-libraries

mohammedhashim790 commented 1 year ago

https://github.com/flet-dev/serious-python/tree/main/src/serious_python#adding-custom-python-libraries

Hey!. Thanks for the information. I followed the steps that's mentioned in the link, yet I get this issue while building numpy using toolchain.

I tried several solution to resolve Cython dependency issue, but none worked. Have you got any solution to this problem?

Any help would be greatly appreciated. Thanks!

[DEBUG   ] x numpy-1.24.2/versioneer.py
[DEBUG   ] New State: numpy.extract at 2023-11-07 12:36:56.515058
[INFO    ] Install_hostpython_prerequisites numpy
[INFO    ] Executing pip for hostpython with: ['install', 'Cython==0.29.36']
[INFO    ] Running Shell: /Users/hashim/Downloads/kivy-ios/dist/hostpython3/bin/pip3 ('install', 'Cython==0.29.36') {'_iter': True, '_out_bufsize': 1, '_err_to_out': True}
[DEBUG   ] WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
[DEBUG   ] WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/cython/
[DEBUG   ] WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/cython/
[DEBUG   ] WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/cython/
[DEBUG   ] WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/cython/
[DEBUG   ] WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/cython/
[DEBUG   ] Could not fetch URL https://pypi.org/simple/cython/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/cython/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
[DEBUG   ] ERROR: Could not find a version that satisfies the requirement Cython==0.29.36 (from versions: none)
[DEBUG   ] ERROR: No matching distribution found for Cython==0.29.36
[DEBUG   ] WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
[DEBUG   ] Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
[DEBUG   ] WARNING: There was an error checking the latest version of pip.
[DEBUG   ] 
Exception in thread background thread for pid 96761:
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.11/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/opt/homebrew/Cellar/python@3.11/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/homebrew/lib/python3.11/site-packages/sh-1.14.3-py3.11.egg/sh.py", line 1641, in wrap
    fn(*rgs, **kwargs)
  File "/opt/homebrew/lib/python3.11/site-packages/sh-1.14.3-py3.11.egg/sh.py", line 2569, in background_thread
    handle_exit_code(exit_code)
  File "/opt/homebrew/lib/python3.11/site-packages/sh-1.14.3-py3.11.egg/sh.py", line 2269, in fn
    return self.command.handle_command_exit_code(exit_code)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/sh-1.14.3-py3.11.egg/sh.py", line 869, in handle_command_exit_code
    raise exc
sh.ErrorReturnCode_1: 

  RAN: /Users/hashim/Downloads/kivy-ios/dist/hostpython3/bin/pip3 install Cython==0.29.36

  STDOUT:
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/cython/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/cython/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS... (1553 more, please see e.stdout)

  STDERR:

Traceback (most recent call last):
  File "/opt/homebrew/bin/toolchain", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/kivy_ios/toolchain.py", line 1562, in main
    ToolchainCL()
  File "/opt/homebrew/lib/python3.11/site-packages/kivy_ios/toolchain.py", line 1306, in __init__
    getattr(self, args.command)()
  File "/opt/homebrew/lib/python3.11/site-packages/kivy_ios/toolchain.py", line 1375, in build
    build_recipes(args.recipe, ctx)
  File "/opt/homebrew/lib/python3.11/site-packages/kivy_ios/toolchain.py", line 1140, in build_recipes
    recipe.execute()
  File "/opt/homebrew/lib/python3.11/site-packages/kivy_ios/toolchain.py", line 701, in execute
    self.install_hostpython_prerequisites()
  File "/opt/homebrew/lib/python3.11/site-packages/kivy_ios/toolchain.py", line 78, in _cache_execution
    f(self, *args, **kwargs)
  File "/opt/homebrew/lib/python3.11/site-packages/kivy_ios/toolchain.py", line 766, in install_hostpython_prerequisites
    _hostpython_pip(["install", prerequisite])
  File "/opt/homebrew/lib/python3.11/site-packages/kivy_ios/toolchain.py", line 1201, in _hostpython_pip
    shprint(pip_cmd, *args)
  File "/opt/homebrew/lib/python3.11/site-packages/kivy_ios/toolchain.py", line 60, in shprint
    for line in cmd:
  File "/opt/homebrew/lib/python3.11/site-packages/sh-1.14.3-py3.11.egg/sh.py", line 915, in next
    self.wait()
  File "/opt/homebrew/lib/python3.11/site-packages/sh-1.14.3-py3.11.egg/sh.py", line 845, in wait
    self.handle_command_exit_code(exit_code)
  File "/opt/homebrew/lib/python3.11/site-packages/sh-1.14.3-py3.11.egg/sh.py", line 869, in handle_command_exit_code
    raise exc
sh.ErrorReturnCode_1: 

  RAN: /Users/hashim/Downloads/kivy-ios/dist/hostpython3/bin/pip3 install Cython==0.29.36

  STDOUT:
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/cython/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/cython/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS... (1553 more, please see e.stdout)

  STDERR:

Similarly, There's JNUI issue while building numpy from p4a.

[INFO]:    Trying first build of pyjnius to get cython files: this is expected to fail                                                                                                                                              [825/1978]
[DEBUG]:   -> running python3 setup.py build_ext -v
[DEBUG]:        Traceback (most recent call last):
[DEBUG]:          File "/storage0/naspi/metameta/mtag/.buildozer/android/platform/build-arm64-v8a_armeabi-v7a/build/other_builds/pyjnius-sdl2/armeabi-v7a__ndk_target_21/pyjnius/setup.py", line 70, in <module>
[DEBUG]:            assert JAVA.is_jdk(), "You need a JDK, we only found a JRE. Try setting JAVA_HOME"
[DEBUG]:        AssertionError: You need a JDK, we only found a JRE. Try setting JAVA_HOME
Exception in thread background thread for pid 11857:
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/zeed/.local/lib/python3.10/py/lib/python3.10/site-packages/sh.py", line 1641, in wrap
    fn(*rgs, **kwargs)
  File "/home/zeed/.local/lib/python3.10/py/lib/python3.10/site-packages/sh.py", line 2569, in background_thread
    handle_exit_code(exit_code)
  File "/home/zeed/.local/lib/python3.10/py/lib/python3.10/site-packages/sh.py", line 2269, in fn
    return self.command.handle_command_exit_code(exit_code)
  File "/home/zeed/.local/lib/python3.10/py/lib/python3.10/site-packages/sh.py", line 869, in handle_command_exit_code
    raise exc
sh.ErrorReturnCode_1:

  RAN: /storage0/naspi/metameta/mtag/.buildozer/android/platform/build-arm64-v8a_armeabi-v7a/build/other_builds/hostpython3/desktop/hostpython3/native-build/python3 setup.py build_ext -v

  STDOUT:
Traceback (most recent call last):
  File "/storage0/naspi/metameta/mtag/.buildozer/android/platform/build-arm64-v8a_armeabi-v7a/build/other_builds/pyjnius-sdl2/armeabi-v7a__ndk_target_21/pyjnius/setup.py", line 70, in <module>
    assert JAVA.is_jdk(), "You need a JDK, we only found a JRE. Try setting JAVA_HOME"
AssertionError: You need a JDK, we only found a JRE. Try setting JAVA_HOME

  STDERR:

[INFO]:    pyjnius first build failed (as expected)
[INFO]:    Running cython where appropriate
[INFO]:    Cythonize jnius/jnius.pyx
[DEBUG]:   -> running python3 -cimport sys; from Cython.Compiler.Main import setuptools_main; sys.exit(setuptools_main()); ./jnius/jnius.pyx
[DEBUG]:        /home/zeed/.local/lib/python3.10/py/lib/python3.10/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /sto
rage0/naspi/metameta/mtag/.buildozer/android/platform/build-arm64-v8a_armeabi-v7a/build/other_builds/pyjnius-sdl2/armeabi-v7a__ndk_target_21/pyjnius/jnius/jnius.pyx
[DEBUG]:          tree = Parsing.p_module(s, pxd, full_module_name)
[DEBUG]:
[DEBUG]:        Error compiling Cython file:
[DEBUG]:        ------------------------------------------------------------
[DEBUG]:        ...
[DEBUG]:        import sys
[DEBUG]:        import traceback
[DEBUG]:
[DEBUG]:        include "jnius_compat.pxi"
[DEBUG]:        include "jni.pxi"
[DEBUG]:        include "config.pxi"
[DEBUG]:        ^
[DEBUG]:        ------------------------------------------------------------
[DEBUG]:
[DEBUG]:        jnius/jnius.pyx:100:0: 'config.pxi' not found
[DEBUG]:
[DEBUG]:        Error compiling Cython file:
[DEBUG]:        ------------------------------------------------------------
[DEBUG]:        ...
[DEBUG]:
[DEBUG]:        include "jnius_compat.pxi"
[DEBUG]:        include "jni.pxi"
[DEBUG]:        include "config.pxi"
[DEBUG]:
[DEBUG]:        IF JNIUS_PLATFORM == "android":
[DEBUG]:          ^
[DEBUG]:        ------------------------------------------------------------
[DEBUG]:
[DEBUG]:        jnius/jnius.pyx:102:3: Compile-time name 'JNIUS_PLATFORM' not defined
FeodorFitsner commented 1 year ago

Found this: https://stackoverflow.com/a/63084099/1435891 - could be your case.