search

kivy / python-for-android

Turn your Python application into an Android APK
https://python-for-android.readthedocs.io
MIT License
8.1k stars 1.81k forks source link

load `ssl` and `crypto` libs before loading python, for (4.1.2 or ear… #903

Closed akshayaurora closed 6 years ago

akshayaurora commented 7 years ago

…lier devices)

closes #866

dkruk commented 7 years ago

I tried it on Android Virtual Device and unfortunately it doesn't work for me. I used Android 4.1(API=16) and Android 4.2(API=17) for checking.

inclement commented 7 years ago

@akshayaurora I feel that I lost track of this issue - is this still a good fix for a current bug?

rnixx commented 7 years ago

should cffi be considered here as well? https://github.com/kivy/python-for-android/pull/903/files#diff-58dede73f12c2d1e35c61afbe5f187aaR25

rnixx commented 7 years ago

and what about other potential libs? does it make sence to have auto detection of everything contained in lib/armeabi-v7a/ in the apk?

akshayaurora commented 7 years ago

@inclement yes, this is a working fix for the current issue on older androids.

@rnixx I would not be comfortable with loading all available libs in that folder... that would mean malicious could be easily injected... is there a easier workaround for this?

cffi could definitely be included here https://github.com/kivy/python-for-android/pull/903/files#diff-58dede73f12c2d1e35c61afbe5f187aaR25

akshayaurora commented 7 years ago

@dzmitrykrukouski the app would crash on emulator cause of missing OpenGL... please test on actual device.

rnixx commented 7 years ago

@akshayaurora It would be great if libs to load get extracted from installed recipes if there are extensions. Whether there is something to load can be defined on the related recipes. It might be just a text file generated somewhere and read in https://github.com/kivy/python-for-android/blob/master/pythonforandroid/bootstraps/sdl2/build/src/org/kivy/android/PythonUtil.java.

What do you think?

Btw - libs are loaded in https://github.com/kivy/python-for-android/blob/master/pythonforandroid/bootstraps/webview/build/src/org/kivy/android/PythonUtil.java as well. Maybe the final mechanism needs to be used here too.

rnixx commented 7 years ago

There's one more https://github.com/kivy/python-for-android/blob/master/pythonforandroid/bootstraps/service_only/build/src/org/kivy/android/PythonUtil.java

dkruk commented 7 years ago

@akshayaurora unfortunately I don't have actual device with Android 4.2 or earlier.

rnixx commented 7 years ago

Some updates

akshayaurora commented 7 years ago

@rnixx could you please put up a example that uses needs, uses cffi? I could test with including it.

akshayaurora commented 7 years ago

loading python2.7 failed as well - again with some symbols not found.

@rnixx could you paste that log, would be interesting to see which symbols were not found.

rnixx commented 7 years ago

@akshayaurora ad example -> just add python2, cryptography and pyopenssl to your recipes.

When starting App which has been built with NDK r13b and commented out python3.5m here https://github.com/kivy/python-for-android/compare/master...rnixx:master#diff-58dede73f12c2d1e35c61afbe5f187aaR42, adb logcat tells:

V/PythonActivity( 6245): About to do super onCreate V/SDL ( 6245): Device: maguro V/SDL ( 6245): Model: Galaxy Nexus V/SDL ( 6245): onCreate():null D/dalvikvm( 6245): Trying to load lib /data/app-lib/org.foo.myapp/libSDL2.so 0x4185c720 D/dalvikvm( 6245): Added shared lib /data/app-lib/org.foo.myapp/libSDL2.so 0x4185c720 D/dalvikvm( 6245): Trying to load lib /data/app-lib/org.foo.myapp/libSDL2_image.so 0x4185c720 D/dalvikvm( 6245): Added shared lib /data/app-lib/org.foo.myapp/libSDL2_image.so 0x4185c720 D/dalvikvm( 6245): No JNI_OnLoad found in /data/app-lib/org.foo.myapp/libSDL2_image.so 0x4185c720, skipping init D/dalvikvm( 6245): Trying to load lib /data/app-lib/org.foo.myapp/libSDL2_mixer.so 0x4185c720 D/dalvikvm( 6245): Added shared lib /data/app-lib/org.foo.myapp/libSDL2_mixer.so 0x4185c720 D/dalvikvm( 6245): No JNI_OnLoad found in /data/app-lib/org.foo.myapp/libSDL2_mixer.so 0x4185c720, skipping init D/dalvikvm( 6245): Trying to load lib /data/app-lib/org.foo.myapp/libSDL2_ttf.so 0x4185c720 D/dalvikvm( 6245): Added shared lib /data/app-lib/org.foo.myapp/libSDL2_ttf.so 0x4185c720 D/dalvikvm( 6245): No JNI_OnLoad found in /data/app-lib/org.foo.myapp/libSDL2_ttf.so 0x4185c720, skipping init D/dalvikvm( 6245): Trying to load lib /data/app-lib/org.foo.myapp/libssl1.0.2h.so 0x4185c720 D/dalvikvm( 6245): Added shared lib /data/app-lib/org.foo.myapp/libssl1.0.2h.so 0x4185c720 D/dalvikvm( 6245): No JNI_OnLoad found in /data/app-lib/org.foo.myapp/libssl1.0.2h.so 0x4185c720, skipping init D/dalvikvm( 6245): Trying to load lib /data/app-lib/org.foo.myapp/libcrypto1.0.2h.so 0x4185c720 D/dalvikvm( 6245): Added shared lib /data/app-lib/org.foo.myapp/libcrypto1.0.2h.so 0x4185c720 D/dalvikvm( 6245): No JNI_OnLoad found in /data/app-lib/org.foo.myapp/libcrypto1.0.2h.so 0x4185c720, skipping init D/dalvikvm( 6245): Trying to load lib /data/app-lib/org.foo.myapp/libffi.so 0x4185c720 D/dalvikvm( 6245): Added shared lib /data/app-lib/org.foo.myapp/libffi.so 0x4185c720 D/dalvikvm( 6245): No JNI_OnLoad found in /data/app-lib/org.foo.myapp/libffi.so 0x4185c720, skipping init D/dalvikvm( 6245): Trying to load lib /data/app-lib/org.foo.myapp/libpython2.7.so 0x4185c720 E/dalvikvm( 6245): dlopen("/data/app-lib/org.foo.myapp/libpython2.7.so") failed: Cannot load library: soinfo_relocate(linker.cpp:975): cannot locate symbol "wait4" referenced by "libpython2.7.so"... D/dalvikvm( 6245): Trying to load lib /data/app-lib/org.foo.myapp/libmain.so 0x4185c720 E/dalvikvm( 6245): dlopen("/data/app-lib/org.foo.myapp/libmain.so") failed: Cannot load library: soinfo_link_image(linker.cpp:1635): could not load library "libpython2.7.so" needed by "libmain.so"; caused by find_library(linker.cpp:889): "libpython2.7.so" failed to load previously W/System.err( 6245): Cannot load library: soinfo_link_image(linker.cpp:1635): could not load library "libpython2.7.so" needed by "libmain.so"; caused by find_library(linker.cpp:889): "libpython2.7.so" failed to load previously

As said above, when leaving python3.5m in the list I get the error message that this one is not found

rnixx commented 7 years ago

This is the error when building with NDK r9c NOT trying to load _cffi_backend.so manually like so https://github.com/kivy/python-for-android/compare/master...rnixx:master#diff-58dede73f12c2d1e35c61afbe5f187aaR80:

I/python ( 6652): File "/home/me/workspace/myapp/.buildozer/android/app/myapp/messaging.py", line 22, in I/python ( 6652): File "/home/me/workspace/myapp/.buildozer/android/app/twisted/internet/ssl.py", line 59, in I/python ( 6652): File "/home/me/workspace/myapp/.buildozer/android/platform/build/dists/myapp/private/lib/python2.7/site-packages/OpenSSL/init.py", line 8, in I/python ( 6652): File "/home/me/workspace/myapp/.buildozer/android/platform/build/dists/myapp/private/lib/python2.7/site-packages/OpenSSL/rand.py", line 11, in I/python ( 6652): File "/home/me/workspace/myapp/.buildozer/android/platform/build/dists/myapp/private/lib/python2.7/site-packages/OpenSSL/_util.py", line 3, in I/python ( 6652): File "/home/me/workspace/myapp/.buildozer/android/platform/build/dists/myapp/private/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 14, in I/python ( 6652): ImportError: Cannot load library: find_library(linker.cpp:889): "/data/data/at.r2solutions.r2_client/files/app/lib/python2.7/site-packages/_cffi_backend.so" failed to load previously

You need to import from OpenSSL import SSL to provoke this.

rnixx commented 7 years ago

And this happens if I try to load _cffi_backend.so manually in PythonUtil.java at startup:

D/dalvikvm( 7370): Trying to load lib /data/data/org.foo.myapp/files/app/lib/python2.7/site-packages/_cffi_backend.so 0x4185c8f8 E/dalvikvm( 7370): dlopen("/data/data/org.foo.myapp/files/app/lib/python2.7/site-packages/_cffi_backend.so") failed: Cannot load library: soinfo_relocate(linker.cpp:975): cannot locate symbol "getline" referenced by "_cffi_backend.so"...

rnixx commented 7 years ago

@akshayaurora Do you need more / other details?

akshayaurora commented 7 years ago

@rnixx nope, that's enough details...I think I have seen the getline issue in a older version...searching...

akshayaurora commented 7 years ago

just for reference http://stackoverflow.com/questions/13112784/undefined-reference-to-getline-in-c

frankgould commented 7 years ago

I was able to resolve the following error (pasted below) by updating 5 copies of PythonUtil.java after searching for them in .buildozer. I inserted the two lines from the github link below the error, in these 5 files and my Android 4.2.2 tablet ran the app flawlessly. I have had this problem since September last year and am very happy KeyWeeUsr was able to guide to the solution, thank you very much.

E/dalvikvm( 3869): dlopen("/data/app-lib/com.masterpics.slideshow-1/libpython2.7.so") failed: Cannot load library: soinfo_link_image(linker.cpp:1635): could not load library "libssl1.0.2h.so" needed by "libpython2.7.so"; caused by load_library(linker.cpp:745): library "libssl1.0.2h.so" not found

https://github.com/kivy/python-for-android/pull/903/commits/c1659b8fc195705d83cef55a7020ebce60d02958

I probably should go back to see if I can remove this code above to see if it is unnecessary in some of the PythonUtil.java files. Below are the list I updated: .buildozer/android/platform/build/build/bootstrap_builds/sdl2-python2/src/org/kivy/android .buildozer/android/platform/build/dists/slideshow/src/org/kivy/android .buildozer/android/platform/python-for-android-master/pythonforandroid/bootstraps/sdl2/build/src/org/kivy/android .buildozer/android/platform/python-for-android-master/pythonforandroid/bootstraps/service_only/build/src/org/kivy/android .buildozer/android/platform/python-for-android-master/pythonforandroid/bootstraps/webview/build/src/org/kivy/android

Thank you all!

Zen-CODE commented 6 years ago

Using this path, I get:

SDL Error.
Error: Couldn't load python3.5m from load dalvik.system.PathClassLoader[DexPathList[[zip_file "/data/app/xxx.apk"],nativeLibraryDirectories=[/data/app-lib/xxx, /vendor/lib, /system/lib]]]:
findLibrary returned Null

If I comment out the python3.5m from line 18 of the PR, I get

SDL Error
And error occurred while trying to start the application. Please try again and/or re-install
Error: Cannot load library: soinfo_link_image(linker.cpp:1635): could not load library "libpython2.7.so" needed by "main.so"; caused by find_library(linker.cpp:889): "libpython2.7.so" filed to load previously.

We are using 2.7, and the APK works on Andorid > 4.4.

inclement commented 6 years ago

@Zen-CODE I've just come around to investigating this, and it's an annoyance that our library loader actually obscures the real errors. If you're still looking at it, could you change the PythonActivity code to:

    public static void loadLibraries(File filesDir) {

        String filesDirPath = filesDir.getAbsolutePath();
        boolean foundPython = false;

        for (String lib : getLibraries()) {
            Log.v("python", "Loading library: " + lib);
            try {
                System.loadLibrary(lib);
                if (lib.startsWith("python")) {
                    foundPython = true;
                }
            } catch(UnsatisfiedLinkError e) {
                // If this is the last possible libpython
                // load, and it has failed, give a more
                // general error
                Log.v("python", "Library loading error: " + e.getMessage());
                if (lib.startsWith("python3.6") && !foundPython) {
                    throw new java.lang.RuntimeException("Could not load any libpythonXXX.so");
                } else if (lib.startsWith("python")) {
                    continue;
                } else {
                    Log.v("python", "An UnsatisfiedLinkError occurred loading " + lib);
                    throw e;
                }
            }
        }

        try {
            System.load(filesDirPath + "/lib/python2.7/lib-dynload/_io.so");
            System.load(filesDirPath + "/lib/python2.7/lib-dynload/unicodedata.so");
        } catch(UnsatisfiedLinkError e) {
            Log.v(TAG, "Failed to load _io.so or unicodedata.so...but that's okay.");
        }

        try {
            // System.loadLibrary("ctypes");
            System.load(filesDirPath + "/lib/python2.7/lib-dynload/_ctypes.so");
        } catch(UnsatisfiedLinkError e) {
            Log.v(TAG, "Unsatisfied linker when loading ctypes");
        }

        Log.v(TAG, "Loaded everything!");
    }

This should print some more helpful information about what has gone wrong. I think I have it working now, but am not yet sure because it still fails for an emulator-related reason. I did have to work through a couple of unexpected problems, including that building for api 19 didn't work (while 15 appears to).

If I'm right about having it working now, I'll fix up this PR and propose a new one for people to test within the next few days. The method here seems solid though. I'm sorry it's taken so long to get around to it, I haven't tackled it before due to the lack of devices with this issue - I would have merged it if it clearly worked for everyone, but as above it has seemed not quite there.

Zen-CODE commented 6 years ago

Insert the above code in PythonActivity.java still gives me this in the logcat.

V/PythonActivity( 8502): Data version is 1503657024.2
I/SurfaceFlinger(  476): EventThread Client Pid (8502) created
D/dalvikvm( 8502): Trying to load lib /data/app-lib/xxx-1/libSDL2.so 0x4133b4a0
D/dalvikvm( 8502): Added shared lib /data/app-lib/xxx-1/libSDL2.so 0x4133b4a0
D/dalvikvm( 8502): Trying to load lib /data/app-lib/xxx-1/libSDL2_image.so 0x4133b4a0
D/dalvikvm( 8502): Added shared lib /data/app-lib/xxx-1/libSDL2_image.so 0x4133b4a0
D/dalvikvm( 8502): No JNI_OnLoad found in /data/app-lib/xxx-1/libSDL2_image.so 0x4133b4a0, skipping init
D/dalvikvm( 8502): Trying to load lib /data/app-lib/xxx-1/libSDL2_mixer.so 0x4133b4a0
D/dalvikvm( 8502): Added shared lib /data/app-lib/xxx-1/libSDL2_mixer.so 0x4133b4a0
D/dalvikvm( 8502): No JNI_OnLoad found in /data/app-lib/xxx-1/libSDL2_mixer.so 0x4133b4a0, skipping init
D/dalvikvm( 8502): Trying to load lib /data/app-lib/xxx-1/libSDL2_ttf.so 0x4133b4a0
D/dalvikvm( 8502): Added shared lib /data/app-lib/xxx-1/libSDL2_ttf.so 0x4133b4a0
D/dalvikvm( 8502): No JNI_OnLoad found in /data/app-lib/xxx-1/libSDL2_ttf.so 0x4133b4a0, skipping init
D/dalvikvm( 8502): Trying to load lib /data/app-lib/xxx-1/libpython2.7.so 0x4133b4a0
E/dalvikvm( 8502): dlopen("/data/app-lib/xxx-1/libpython2.7.so") failed: Cannot load library: soinfo_link_image(linker.cpp:1635): could not load library "libsqlite3.so" needed by "libpython2.7.so"; caused by load_library(linker.cpp:745): library "libsqlite3.so" not found
W/System.err( 8502): Cannot load library: soinfo_link_image(linker.cpp:1635): could not load library "libsqlite3.so" needed by "libpython2.7.so"; caused by load_library(linker.cpp:745): library "libsqlite3.so" not found
D/skia    ( 8502): Flag is not 10

That being said, I see there are multiple PythonActivity.java files. Which one should be changing?

Zen-CODE commented 6 years ago

Also, building with "android.api = 15" still produces the same error.

inclement commented 6 years ago

@Zen-CODE My best guess is that your error represents the same problem as openssl, but with libsqlite3.so instead, in which case it could be fixed by adding "sqlite3" to the module loading list. I intend to test sqlite3 myself shortly to confirm this.

I'll push my changes to the logging to the master branch shortly, to make it easier to use.

Sorry about being unclear about where to apply the changes, the best thing to do is to clean any existing distributions and edit the version in pythonforandroid/bootstraps/sdl2/build/src/org/kivy/android. I'll try to add some proper documentation about this.

inclement commented 6 years ago

I've just pushed my logging improvements to master, so you should be able to get them by pulling that. I'm now investigating sqlite3.

inclement commented 6 years ago

@Zen-CODE I've made PR https://github.com/kivy/python-for-android/pull/1106 with my rebase plus restructuring. Does that work for you?

Zen-CODE commented 6 years ago

sqlite3 is in the requirements. Get this error on screen:

SDL Error 
Error: cannot load library:
soinfo_link_image(linker.cpp:1635): could not load library "libsqlite3.so" needed by "libpython2.7.so"; caused by load_library (linker.cpp: 745): library "libsqlite3.so" not found.

And the log has this:

I/SurfaceFlinger(  130): EventThread Client Pid (6949) created
D/dalvikvm( 6949): Trying to load lib /data/app-lib/speedtest.maths.camiweb.com-1/libSDL2.so 0x4176f440
D/dalvikvm( 6949): Added shared lib /data/app-lib/speedtest.maths.camiweb.com-1/libSDL2.so 0x4176f440
D/dalvikvm( 6949): Trying to load lib /data/app-lib/speedtest.maths.camiweb.com-1/libSDL2_image.so 0x4176f440
D/dalvikvm( 6949): Added shared lib /data/app-lib/speedtest.maths.camiweb.com-1/libSDL2_image.so 0x4176f440
D/dalvikvm( 6949): No JNI_OnLoad found in /data/app-lib/speedtest.maths.camiweb.com-1/libSDL2_image.so 0x4176f440, skipping init
D/dalvikvm( 6949): Trying to load lib /data/app-lib/speedtest.maths.camiweb.com-1/libSDL2_mixer.so 0x4176f440
D/dalvikvm( 6949): Added shared lib /data/app-lib/speedtest.maths.camiweb.com-1/libSDL2_mixer.so 0x4176f440
D/dalvikvm( 6949): No JNI_OnLoad found in /data/app-lib/speedtest.maths.camiweb.com-1/libSDL2_mixer.so 0x4176f440, skipping init
D/dalvikvm( 6949): Trying to load lib /data/app-lib/speedtest.maths.camiweb.com-1/libSDL2_ttf.so 0x4176f440
D/dalvikvm( 6949): Added shared lib /data/app-lib/speedtest.maths.camiweb.com-1/libSDL2_ttf.so 0x4176f440
D/dalvikvm( 6949): No JNI_OnLoad found in /data/app-lib/speedtest.maths.camiweb.com-1/libSDL2_ttf.so 0x4176f440, skipping init
D/dalvikvm( 6949): Trying to load lib /data/app-lib/speedtest.maths.camiweb.com-1/libpython2.7.so 0x4176f440
E/dalvikvm( 6949): dlopen("/data/app-lib/speedtest.maths.camiweb.com-1/libpython2.7.so") failed: Cannot load library: soinfo_link_image(linker.cpp:1635): could not load library "libsqlite3.so" needed by "libpython2.7.so"; caused by load_library(linker.cpp:745): library "libsqlite3.so" not found
W/System.err( 6949): Cannot load library: soinfo_link_image(linker.cpp:1635): could not load library "libsqlite3.so" needed by "libpython2.7.so"; caused by load_library(linker.cpp:745): library "libsqlite3.so" not found

Complete log here: https://gist.github.com/Zen-CODE/79998bb0a569374dde008ab58b8bc3da

inclement commented 6 years ago

@Zen-CODE Just to be clear, by 'module loading list' I mean the list of strings in PythonUtil.java, not the buildozer.spec requirements.

I believe my PR at https://github.com/kivy/python-for-android/pull/1106 should fix this issue. At the very least, it should give an extended log about what it finds (or fails to find) when trying to load sqlite3.

Zen-CODE commented 6 years ago

The log above was generated using PR #1106 changes.

inclement commented 6 years ago

@Zen-CODE Are you sure it's using the new code? There really should now be more logs than that. Could you paste the full logcat output?

Zen-CODE commented 6 years ago

The log above was generated using PR #1106 changes, but let me do another one.

So, I am changing the file:

/.buildozer/android/platform/python-for-android-master/pythonforandroid/bootstraps/sdl2/build/src/org/kivy/android/PythonUtil.java To verify this is the one being used, I do a clean and build and the (deliberate) syntax error I inserted crashes it, so it is using the file. When I use the new version from PR 1106, it crashes with the same libsqlite3 issue, full output here: https://gist.github.com/Zen-CODE/cdca11cb6e57aab7e2962ffd876597b6
inclement commented 6 years ago

@Zen-CODE I still can't work out why you wouldn't be seeing the more extended logs - in particular, the line at https://github.com/kivy/python-for-android/pull/1106/files#diff-58dede73f12c2d1e35c61afbe5f187aaR58 should be called for every library, printing output using the "pythonutil" tag. Do you see anything I'm missing here (maybe I'm mistaken!).

The actual dist you've built should be in the .buildozer somewhere - I think .buildozer/android/platform/build. Could you check that the PythonUtil in your dist matches the changes in the main file at /.buildozer/android/platform/python-for-android-master/pythonforandroid/bootstraps/sdl2/build/src/org/kivy/android/PythonUtil.java? Assuming so, would you be able to zip up and send me the dist you find there? I think I should be able to import it and try to investigate from there.

Zen-CODE commented 6 years ago

@inclement. Sorry, it just occurred to me that perhaps it the way I capture the log using the "> log.txt" post-fix.

buildozer android debug depoly run logcat > log.txt

Perhaps the logging you are expecting to not redirect? Or how should I generate the log?

inclement commented 6 years ago

I'm closing this as superseded by https://github.com/kivy/python-for-android/pull/1106 (now merged), but I'm still not clear if it has fixed the issue for everyone. Feel free to open a new issue if you still experience this problem with the master branch.