Install wheel package at runtime.

[nitanmarcel]

I'm thinking of adding plugin support to my application via pypi and I was wondering if it's possible to install a wheel package or pip package at runtime and then load it in the env.

Is something like this even possible?

[nitanmarcel]

I tried to use pip as a module and I haven't got far away:

Cmdline: /system/bin/app_process64 /data/user/0/dev.marcelnitan.r2droid/files/chaquopy/AssetFinder/requirements/pip/__pip-runner__.py install --ignore-installed --no-user --prefix /data/user/0/dev.marcelnitan.r2droid/cache/chaquopy/tmp/pip-build-env-asj91hrn/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools>=40.8.0
2024-04-25 19:32:39.669 26906-26906 python.stdout           dev.marcelnitan.r2droid              I    Installing build dependencies: finished with status 'error'
2024-04-25 19:32:39.680 26906-26906 python.stderr           dev.marcelnitan.r2droid              W    error: subprocess-exited-with-error
2024-04-25 19:32:39.680 26906-26906 python.stderr           dev.marcelnitan.r2droid              W    
2024-04-25 19:32:39.680 26906-26906 python.stderr           dev.marcelnitan.r2droid              W    × pip subprocess to install build dependencies did not run successfully.
2024-04-25 19:32:39.680 26906-26906 python.stderr           dev.marcelnitan.r2droid              W    │ exit code: -6
2024-04-25 19:32:39.680 26906-26906 python.stderr           dev.marcelnitan.r2droid              W    ╰─> [0 lines of output]
2024-04-25 19:32:39.681 26906-26906 python.stderr           dev.marcelnitan.r2droid              W        [end of output]
2024-04-25 19:32:39.681 26906-26906 python.stderr           dev.marcelnitan.r2droid              W    
2024-04-25 19:32:39.681 26906-26906 python.stderr           dev.marcelnitan.r2droid              W    note: This error originates from a subprocess, and is likely not a problem with pip.
2024-04-25 19:32:39.684 26906-26906 python.stderr           dev.marcelnitan.r2droid              W  error: subprocess-exited-with-error
2024-04-25 19:32:39.684 26906-26906 python.stderr           dev.marcelnitan.r2droid              W  
2024-04-25 19:32:39.685 26906-26906 python.stderr           dev.marcelnitan.r2droid              W  × pip subprocess to install build dependencies did not run successfully.
2024-04-25 19:32:39.685 26906-26906 python.stderr           dev.marcelnitan.r2droid              W  │ exit code: -6
2024-04-25 19:32:39.685 26906-26906 python.stderr           dev.marcelnitan.r2droid              W  ╰─> See above for output.
2024-04-25 19:32:39.685 26906-26906 python.stderr           dev.marcelnitan.r2droid              W  
2024-04-25 19:32:39.685 26906-26906 python.stderr           dev.marcelnitan.r2droid              W  note: This error originates from a subprocess, and is likely not a problem with pip.
2024-04-25 19:32:39.719 26906-26925 AdrenoGLES-0            dev.marcelnitan.r2droid              I  QUALCOMM build                   : 8e5405b, I57aaec3440
                                                                                                    Build Date                       : 05/21/21
                                                                                                    OpenGL ES Shader Compiler Version: EV031.32.02.10
                                                                                                    Local Branch                     : mybranchebba1dbe-451b-f160-ac81-1458d0b52ae8
                                                                                                    Remote Branch                    : quic/gfx-adreno.lnx.1.0.r135-rel
                                                                                                    Remote Branch                    : NONE
                                                                                                    Reconstruct Branch               : NOTHING
2024-04-25 19:32:39.719 26906-26925 AdrenoGLES-0            dev.marcelnitan.r2droid              I  Build Config                     : S P 10.0.7 AArch64
2024-04-25 19:32:39.719 26906-26925 AdrenoGLES-0            dev.marcelnitan.r2droid              I  Driver Path                      : /vendor/lib64/egl/libGLESv2_adreno.so
2024-04-25 19:32:39.722 26906-26925 AdrenoGLES-0            dev.marcelnitan.r2droid              I  PFP: 0x016ee190, ME: 0x00000000
2024-04-25 19:32:39.735 26906-26987 Gralloc4                dev.marcelnitan.r2droid              I  mapper 4.x is not supported
2024-04-25 19:32:39.735 26906-26987 Gralloc3                dev.marcelnitan.r2droid              W  mapper 3.x is not supported
2024-04-25 19:32:39.801 26906-26906 Choreographer           dev.marcelnitan.r2droid              I  Skipped 149 frames!  The application may be doing too much work on its main thread.
2024-04-25 19:32:39.832 26906-26991 ProfileInstaller        dev.marcelnitan.r2droid              D  Installing profile for dev.marcelnitan.r2droid
2024-04-25 19:32:52.123 26906-26916 elnitan.r2droid         dev.marcelnitan.r2droid              I  Background concurrent copying GC freed 43398(3143KB) AllocSpace objects, 5(100KB) LOS objects, 49% free, 4678KB/9356KB, paused 156us,35us total 119.557ms

[mhsmith]

See https://github.com/chaquo/chaquopy/issues/623#issuecomment-2079196964.

[nitanmarcel]

See #623 (comment).

Thanks.

I understood in the end that I won't really be able to use pip. But I've found a nice open source alternative called plz that I'm trying to compile and modify

https://github.com/juancarlospaco/plz

It still needs python to run setup.py unfortunately

[nitanmarcel]

Oh and we can build python for Android. This is neat

https://github.com/chaquo/chaquopy/blob/master/target/standalone-python.sh

[mhsmith]

I've found a nice open source alternative called plz that I'm trying to compile and modify

I'm sure there must be a pure-Python alternative that doesn't need to be compiled.

It still needs python to run setup.py unfortunately

If all your packages are available as wheels, then that shouldn't be a problem.

https://github.com/chaquo/chaquopy/blob/master/target/standalone-python.sh

I haven't tested that script for a long time, so it probably doesn't work with the current version of Chaquopy. It was also written before Android started to block apps from including executables (#605).

[nitanmarcel]

I've tried to search one but I couldn't find any so far, I'll probably keep looking. Yeah, will have my own repo with wheels build by the scripts.

[nitanmarcel]

Maybe just seuptools is enough

https://pypi.org/project/setuptools/

[nitanmarcel]

Ah setuptools might be enough

https://github.com/pypa/easy_install/blob/master/easy_install/__init__.py

[nitanmarcel]

For reference I've also found this

https://github.com/pypa/setuptools/blob/0156e248e777eebff9250d83603611585d0c8f12/setuptools/sandbox.py#L247

[nitanmarcel]

looks like I found what I need for wheel files

https://installer.pypa.io/en/stable/

[nitanmarcel]

The files directory can't be accessed from python side?

PermissionError: [Errno 13] Permission denied: '/data/user/0/dev.marcelnitan.r2droid/files/

[mhsmith]

It can certainly be read and written, but maybe this error comes from doing something else. If you want help, please post the full stack trace and the relevant sections of your code.

[nitanmarcel]

It can certainly be read and written, but maybe this error comes from doing something else. If you want help, please post the full stack trace and the relevant sections of your code.

The issue seems to be only with opening the wheel, but not writing files in the directory. This could be worked around by using extenal cache/files directory where it works fine as the source of the wheel file. Could be an installer thing.

[nitanmarcel]

I'm trying to figure out the sysconfig.get_paths() for chaquopy.

So far I've created these but I'm not 100% i'm using the right paths

PATHS = {
    'data': f"{PREFIX}/usr",
    'include': f"{PREFIX}/usr/include",
    'platinclude': f"{PREFIX}/usr/include/python3.11",
    "platlib": f"{PREFIX}/chaquopy/AssetFinder/requirements/chaquopy_radare",
    'platstdlib': f"{PREFIX}/usr/include",
    'purelib': f"{PREFIX}/chaquopy/AssetFinder/requirements/chaquopy_radare",
    'scripts': f"{PREFIX}/bin",
    'stdlib': f"{PREFIX}/usr/include/python3.11"
}

[nitanmarcel]

It can certainly be read and written, but maybe this error comes from doing something else. If you want help, please post the full stack trace and the relevant sections of your code.

14:00:14.425 18328 18328 E AndroidRuntime: com.chaquo.python.PyException: PermissionError: [Errno 13] Permission denied: '/data/data/dev.marcelnitan.r2droid/files/chaquopy_radare2-5.9.0-0-py3-none-android_21_arm64_v8a.whl'
04-27 14:00:14.425 18328 18328 E AndroidRuntime:    at <python>.zipfile.__init__(zipfile.py:1284)
04-27 14:00:14.425 18328 18328 E AndroidRuntime:    at <python>.installer.sources.open(sources.py:162)
04-27 14:00:14.425 18328 18328 E AndroidRuntime:    at <python>.contextlib.__enter__(contextlib.py:137)
04-27 14:00:14.425 18328 18328 E AndroidRuntime:    at <python>.init.install(init.py:84)
04-27 14:00:14.425 18328 18328 E AndroidRuntime:    at <python>.init.<module>(init.py:116)
04-27 14:00:14.425 18328 18328 E AndroidRuntime:    at <python>.importlib._bootstrap._call_with_frames_removed(<frozen importlib._bootstrap>:241)
04-27 14:00:14.425 18328 18328 E AndroidRuntime:    at <python>.importlib._bootstrap_external.exec_module(<frozen importlib._bootstrap_external>:940)
04-27 14:00:14.425 18328 18328 E AndroidRuntime:    at <python>.java.android.importer.exec_module(importer.py:634)
04-27 14:00:14.425 18328 18328 E AndroidRuntime:    at <python>.java.android.importer.exec_module(importer.py:721)
04-27 14:00:14.425 18328 18328 E AndroidRuntime:    at <python>.importlib._bootstrap._load_unlocked(<frozen importlib._bootstrap>:690)
04-27 14:00:14.425 18328 18328 E AndroidRuntime:    at <python>.importlib._bootstrap._find_and_load_unlocked(<frozen importlib._bootstrap>:1147)
04-27 14:00:14.425 18328 18328 E AndroidRuntime:    at <python>.importlib._bootstrap._find_and_load(<frozen importlib._bootstrap>:1176)
04-27 14:00:14.425 18328 18328 E AndroidRuntime:    at <python>.importlib._bootstrap._gcd_import(<frozen importlib._bootstrap>:1204)
04-27 14:00:14.425 18328 18328 E AndroidRuntime:    at <python>.importlib.import_module(__init__.py:126)
04-27 14:00:14.425 18328 18328 E AndroidRuntime:    at <python>.chaquopy_java.Java_com_chaquo_python_Python_getModuleNative(chaquopy_java.pyx:129)
04-27 14:00:14.425 18328 18328 E AndroidRuntime:    at com.chaquo.python.Python.getModuleNative(Native Method)
04-27 14:00:14.425 18328 18328 E AndroidRuntime:    at com.chaquo.python.Python.getModule(Python.java:84)
04-27 14:00:14.425 18328 18328 E AndroidRuntime:    at dev.marcelnitan.r2droid.tabs.HomeTab.Content(HomeTab.kt:48)
04-27 14:00:14.425 18328 18328 E AndroidRuntime:    at cafe.adriel.voyager.navigator.tab.TabKt$CurrentTab$1.invoke(Tab.kt:13)
04-27 14:00:14.425 18328 18328 E AndroidRuntime:    at cafe.adriel.voyager.navigator.tab.TabKt$CurrentTab$1.invoke(Tab.kt:12)
04-27 14:00:14.425 18328 18328 E AndroidRuntime:    at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:109)

[nitanmarcel]

This is the wheel installer script I have so far that can be called from java:

# CREDITS: https://github.com/python-poetry/poetry/blob/ef75e7c85508cca498be7c630d8373a5c0b26586/src/poetry/installation/wheel_installer.py

from __future__ import annotations

import logging

from pathlib import Path
from typing import TYPE_CHECKING

from installer import install
from installer.destinations import SchemeDictionaryDestination
from installer.sources import WheelFile
from installer.sources import _WheelFileValidationError

logger = logging.getLogger(__name__)

if TYPE_CHECKING:
    from collections.abc import Collection
    from typing import BinaryIO

    from installer.records import RecordEntry
    from installer.utils import Scheme

class WheelDestination(SchemeDictionaryDestination):
    """ """

    def write_to_fs(
            self,
            scheme: Scheme,
            path: str,
            stream: BinaryIO,
            is_executable: bool,
    ) -> RecordEntry:
        from installer.records import Hash
        from installer.records import RecordEntry
        from installer.utils import copyfileobj_with_hashing
        from installer.utils import make_file_executable

        target_path = Path(self.scheme_dict[scheme]) / path
        if target_path.exists():
            # Contrary to the base library we don't raise an error here since it can
            # break pkgutil-style and pkg_resource-style namespace packages.
            logger.warning(f"Installing {target_path} over existing file")

        parent_folder = target_path.parent
        if not parent_folder.exists():
            # Due to the parallel installation it can happen
            # that two threads try to create the directory.
            parent_folder.mkdir(parents=True, exist_ok=True)

        with target_path.open("wb") as f:
            hash_, size = copyfileobj_with_hashing(stream, f, self.hash_algorithm)

        if is_executable:
            make_file_executable(target_path)

        return RecordEntry(path, Hash(self.hash_algorithm, hash_), size)

class WheelInstaller:
    def __init__(self, prefix: Path) -> None:
        self._bytecode_optimization_levels: Collection[int] = ()
        self.invalid_wheels: dict[Path, list[str]] = {}

        self.paths = {
            'data': f"{prefix}/usr",
            'include': f"{prefix}/usr/include",
            'platinclude': f"{prefix}/usr/include/python3.11",
            "platlib": f"{prefix}/chaquopy/AssetFinder/requirements/chaquopy_radare",
            'platstdlib': f"{prefix}/usr/include",
            'purelib': f"{prefix}/chaquopy/AssetFinder/requirements/chaquopy_radare",
            'scripts': f"{prefix}/bin",
            'stdlib': f"{prefix}/usr/include/python3.11"
        }

    def enable_bytecode_compilation(self, enable: bool = True) -> None:
        self._bytecode_optimization_levels = (-1,) if enable else ()

    def install(self, wheel: Path, interpreter: str = "str") -> None:
        with WheelFile.open(wheel) as source:
            try:
                # Content validation is temporarily disabled because of
                # pypa/installer's out of memory issues with big wheels. See
                # https://github.com/python-poetry/poetry/issues/7983
                source.validate_record(validate_contents=False)
            except _WheelFileValidationError as e:
                self.invalid_wheels[wheel] = e.issues

            scheme_dict = self.paths.copy()
            scheme_dict["headers"] = str(
                Path(scheme_dict["include"]) / source.distribution
            )
            destination = WheelDestination(
                scheme_dict,
                interpreter=interpreter,
                script_kind="posix",
                bytecode_optimization_levels=self._bytecode_optimization_levels,
            )

            install(
                source=source,
                destination=destination,
                additional_metadata={
                },
            )

[mhsmith]

Are you sure that's actually the files directory of the running app? Did you get it from os.environ["HOME"]?

Or maybe you're somehow creating the wheel file without read permissions.

If you still can't work it out, please post the code which puts the wheel file in that location, and calls install.

[nitanmarcel]

Are you sure that's actually the files directory of the running app? Did you get it from os.environ["HOME"]?

Or maybe you're somehow creating the wheel file without read permissions.

If you still can't work it out, please post the code which puts the wheel file in that location, and calls install.

Yes, I manually placed the file there and pasted the path to the install method in my python code. Well I just made the installation in the python code and just loaded the module to trigger it

[nitanmarcel]

File: chaquopy_radare2-5.9.0-0-py3-none-android_21_arm64_v8a.whl
  Size: 12534276         Blocks: 24488   IO Blocks: 512    regular file
Device: 10301h/66305d    Inode: 4821816  Links: 1 Device type: 0,0
Access: (0640/-rw-r-----)       Uid: (    0/    root)     Gid: (    0/    root)
Access: 2024-04-26 17:33:33.650244432 +0300
Modify: 2024-04-26 17:33:33.750244432 +0300
Change: 2024-04-26 17:33:33.750244432 +0300

[nitanmarcel]

I think the issue is due adb pushing files there, java File can't read them either

[mhsmith]

Strange: when I copy a file using Android Studio's Device Explorer, or with adb root followed by adb push, it gets 666 permissions:

-rw-rw-rw- 1 root    root     8849 2024-04-14 17:10 README.rst

You may need to change the permissions manually by running adb shell chmod.

[nitanmarcel]

Does it even work with the root group and owner? Files written by the app have their own

[mhsmith]

Regardless of who owns it, the third r means it's readable by everyone:

-rw-rw-rw-

But in your example, it's only readable by the root user and group:

-rw-r-----

You can change this with the chmod command:

adb root
adb shell chmod o+r /data/data/dev.marcelnitan.r2droid/files/chaquopy_radare2-5.9.0-0-py3-none-android_21_arm64_v8a.whl

[nitanmarcel]

Ah ok.

Anyway can I get the filesDir path in python if I have the context statically visible in a custom Application class in java?

In short, accessing static classes defined in the java app in python

[nitanmarcel]

Or even better the AssetsDir (python path) from Chaquopy

[mhsmith]

You can get the files directory from os.environ["HOME"], as it says in the documentation.

The asset directory doesn't actually exist at runtime, because assets are read directly from the APK. If you want to access any data files from Python, put them in your Python source code directory as shown here.

By the way, there's no need to quote a GitHub comment when you're replying directly below it, unless you want to reply to multiple points separately as I did here. Unnecessary quoting clutters the page and makes the conversation harder to follow. Just post your reply by itself.