apk not build using crystax NDK

[sumitmadhwani]

I am building apk using crystax NDK. I am following the same instructions as mentioned in python-for-android documentation. I am getting following error for the line from kivy.app import App logs

[KarimReefat]

same issue but with other symbol PyDict_Next in the same module _clock.so ( ImportError: dlopen failed: cannot locate symbol "PyDict_Next" referenced by "_clock.so"... ) my_log.txt buildozer.txt

[FeralBytes]

Also encountering the same error; identical to @KarimReefat

[FeralBytes]

I think it is actually caused by either buildozer or python-for-android; because it is using cython in a python 2.7 virtualenv and yet we are targeting python 3; thus it should be compiled in a python3 virtual environment.

[FeralBytes]

Planning to modify P4A tonight to use either venv or virtualenv3 aka virtualenv but where --python=python3.5 Offending line of code is Line Number 327: https://github.com/kivy/python-for-android/blob/master/pythonforandroid/build.py#L327 Posting the details in case some one wants to try before I get to it tomorrow.

[FeralBytes]

Also of note Buildozer is also guilty of forcing python 2.7, so I wll probably have to modify it too Line Number 546: https://github.com/kivy/buildozer/blob/master/buildozer/__init__.py#L546 @inclement you stated you were having an issue too; and you referenced python 3.6, are you using Crystax to compile?

[inclement]

I've been using only crystax for a while, without issues, so I'm not sure what's wrong. The python3 build is quite idiosyncratic though and needs cleaning up, so it's good to have references to ways it's failing for people.

The bit about using a python2.7 virtualenv for python3 builds needs to be fixed, but actually shouldn't in generally stop the build working, as only pure python modules are built this way and most are py2/py3 compatible in the same source code. I've been building python3 apks for a while without any problems of this nature (actually I forgot about this issue!).

With the recent release of python 3.6, it's become particularly important to fix up the python3 build, and I've been doing some preliminary work on this recently. The overall plan is to actually carry the hostpython and targetpython builds, using the build scripts of crystax, rather than using only the prebuilt binaries. I hope that if this is done well, it should both allow support of multiple python versions (in fact, possible most/all recent ones) and make the build more robust for people on different systems. However, things like this always throw up problems, so it's hard to say exactly what will happen when.

[FeralBytes]

@inclement so do you think it is a recent change kivy clock that may be causing the issue then? I have Python 3.5.2. Can you paste your logcat. Also what exactly should the requirements be at minimum; because I am confused between hostpython3crystax and python3crystax. Also I found this output from buildozer to be odd:

[INFO]:    Downloading hostpython3crystax
[INFO]:    Skipping hostpython3crystax download as no URL is set
[INFO]:    Downloading python2
[INFO]:    -> running mkdir -p /home/wolfrage/0.Projects/EzTunes/src/EzTunes/.buildozer/android/platform/build/packages/python2
[INFO]:    -> directory context /home/wolfrage/0.Projects/EzTunes/src/EzTunes/.buildozer/android/platform/build/packages/python2
[INFO]:    -> running basename http://python.org/ftp/python/2.7.2/Python-2.7.2.tar.bz2
[INFO]:    -> running rm -f .mark-Python-2.7.2.tar.bz2
[INFO]:    Downloading python2 from http://python.org/ftp/python/2.7.2/Python-2.7.2.tar.bz2

And finally what exactly is hostpython doing; I am trying to close a gap in understanding that is why I am asking.

[inclement]

@FeralBytes Probably not, the kivy clock module is just the first cython one that gets imported so normally it's the one that registers as broken even though they all are.

This setup.py file in the python-for-android testapp builds folders should produce a working python3 build. To use it, you'll need to update the ndk-dir path to point to your own local ndk.

You don't need to add hostpython3crystax, although it's a minor bug that it will apparently be allowed alongside python2 in the requirements.

[FeralBytes]

@inclement worked through getting that file setup for my enviroment; and finally got it to build APK; but upon launching the APK on android; it crashes again as soon as kivy is imported with a failure pointing at _clock.so . At this time I am unable to build anything with Kivy.

[FeralBytes]

@inclement even tried with the Kivy Hello World Example.

I/python  ( 3069): main.py
I/python  ( 3069): [WARNING] [Config      ] Older configuration version detected (0 instead of 17)
I/python  ( 3069): [WARNING] [Config      ] Upgrading configuration in progress.
I/python  ( 3069): [INFO   ] [Logger      ] Record log in /data/data/com.feralbytes.EzTunes/files/app/.kivy/logs/kivy_70-01-01_0.txt
I/python  ( 3069): [INFO   ] [Kivy        ] v1.9.2.dev0, git-Unknown, 20170131
I/python  ( 3069): [INFO   ] [Python      ] v3.5.0 (default, Dec 24 2015, 05:56:06) 
I/python  ( 3069): [GCC 5.3 20151204]
I/python  ( 3069):  Traceback (most recent call last):
I/python  ( 3069):    File "main.py", line 3, in <module>
I/python  ( 3069):      from kivy.app import App
I/python  ( 3069):    File "/data/data/com.feralbytes.EzTunes/files/app/crystax_python/site-packages/kivy/app.py", line 319, in <module>
I/python  ( 3069):      from kivy.base import runTouchApp, stopTouchApp
I/python  ( 3069):    File "/data/data/com.feralbytes.EzTunes/files/app/crystax_python/site-packages/kivy/base.py", line 29, in <module>
I/python  ( 3069):      from kivy.clock import Clock
I/python  ( 3069):    File "/data/data/com.feralbytes.EzTunes/files/app/crystax_python/site-packages/kivy/clock.py", line 362, in <module>
I/python  ( 3069):      from kivy._clock import CyClockBase, ClockEvent, FreeClockEvent, \
I/python  ( 3069):  ImportError: dlopen failed: cannot locate symbol "PyDict_Next" referenced by "_clock.so"...
I/python  ( 3069): Python for android ended.

Are you currently able to compile anything with Kivy?

[FeralBytes]

By pushing Kivy way back in GIT I was able to get the error to change to: I/python ( 3379): Traceback (most recent call last): I/python ( 3379): File "main.py", line 3, in I/python ( 3379): from kivy.app import App I/python ( 3379): File "/data/data/com.feralbytes.EzTunes/files/app/crystax_python/site-packages/kivy/app.py", line 319, in I/python ( 3379): from kivy.base import runTouchApp, stopTouchApp I/python ( 3379): File "/data/data/com.feralbytes.EzTunes/files/app/crystax_python/site-packages/kivy/base.py", line 30, in I/python ( 3379): from kivy.event import EventDispatcher I/python ( 3379): File "/data/data/com.feralbytes.EzTunes/files/app/crystax_python/site-packages/kivy/event.py", line 8, in I/python ( 3379): import kivy._event I/python ( 3379): ImportError: dlopen failed: cannot locate symbol "PyObject_GetAttr" referenced by "_event.so"... I/python ( 3379): Python for android ended.

Still working backwards.

[FeralBytes]

So that every one is aware; at this time; I have stopped trying Crystax builds and Py3. I just am using my code under Py2; since my code is compatable; that is except python-osc. So I am now going to either switch to Kivy-OSC of port python-osc 3to2. Either way I believe Crystax builds to be broken at this time. I do not see how to fix it and would need some further help before I even try anything else; already lost too much time to it.

[FeralBytes]

For me crystax builds are now working either using Buildozer or a P4A setup.py script. I am not sure what exactly changed; whether it be on my system or else where; but by starting from a basic (not crystax) build with p4a and slowing adding requirements everything began working. @malverick and @KarimReefat are you still having this issue? Have you tracked it down any further? I recommend trying to clean your python for android or buildozer build, dist, and packages directories.

[sumitmadhwani]

@FeralBytes I was implementing a feature for plyer and at that time apk was not building using android ndk so I was suggested to use crystax. After facing this issue, I somehow made my apk build using android ndk. So I continued my work on that feature and thought of looking into this issue sometime later. Now that you have got crystax builds working fine, if you suggest I will close this issue. Or we should wait for @KarimReefat's response?

[FeralBytes]

Well before closing it; do you know if your Crystax build is working again? If Crystax is again working for you; then yes close it; because something got fixed for all of us; but I don't know what. Though it seems the common fact was going back to regular apk build process and working from there.

[inclement]

Just looking through this issue, does it actually have anything to do with the crystax NDK? Do builds specifically work when not using CrystaX, but fail when using it?

The dlopen error appears to be caused by missing linker arguments under some circumstances, but it depends on the local environment in a way I don't understand. I may try some things at the weekend that could fix it.

[sumitmadhwani]

Yes @inclement , the builds were working fine when not using CrystaX NDK and py3.

[inclement]

Okay. Do python2 builds work with crystax? The python2 and python3 builds are quite different internally, so this would be useful to know.

If it's only the python3 build that fails, I probably have a fix for it, but I have to test some things.

On 10/02/17 19:22, Sumit Madhwani wrote:

Yes @inclement https://github.com/inclement , the builds were working fine when not using CrystaX NDK and py3.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kivy/python-for-android/issues/992#issuecomment-279040414, or mute the thread https://github.com/notifications/unsubscribe-auth/ABNQm8CEkO4rhjeFuYEpt5jhbjdIxDqGks5rbLjcgaJpZM4LsuMc.

[sumitmadhwani]

I have not tested it with py2. I will test it using py2 and will inform you till tomorrow. Will that be fine? If you want it urgently, I will do it now.

[inclement]

No rush at all. Thanks!

On 10/02/17 19:27, Sumit Madhwani wrote:

I have not tested it with py2. I will test it using py2 and will inform you till tomorrow. Will that be fine? If you want it urgently, I will do it now.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kivy/python-for-android/issues/992#issuecomment-279041845, or mute the thread https://github.com/notifications/unsubscribe-auth/ABNQmweXkXSD52TIJHUJmQWS-1KvjEpxks5rbLokgaJpZM4LsuMc.

[FeralBytes]

@inclement for me the regression in being able to build was for more than just Crystax; that is until at least I went all the way back to the old tool chain. Then as I built back up everything was working again. So I really don't know what went wrong at least in my case.

[inclement]

I've just pushed https://github.com/kivy/python-for-android/commit/ed1e1a8dd86d39badc41c39677dc287a94d854ef which should (I hope) fix this problem.

[FeralBytes]

Please do not close this bug; I think I now know the issue. @inclement I think your fix actually just highlighted the issue to me. When building my various setup scripts; (yes I have too many; but I was testing to make sure my code can build many ways) I found the 3.5m reference from your push. I also managed to get a useful java traceback.

I think the real issue is; once you build with crystax all of the libs we depend on: "libSDL2.so", "libpython2.7.so" or "libpython3.5.so" depend on "libcrystax.so". The issue arises when you go back to a non-crystax build; and with enough debugging the error actually shows it's self rather than blowing up on Android as happened to me previously. The fix that I have applied to my setup.py files is to create in the ~/.local/share/python-for-android/ directory a flag file such as: crystax-ndk-10.3.2.flag full path: ~/.local/share/python-for-android/crystax-ndk-10.3.2.flag This allows my same type of builds to be fast but if the build NDK changes to Google's NDK or Crystax from the other then I know because the flag does not match the NDK name. When it changes I execute the commands below to prevent any corrupt ".so" files that depend on libcrystax.so. Please note I need to further test these commands to make sure they are correct; only had enough time to understand the problem not to ensure I caught every detail. So I will post back tomorrow with exact commands and then I can start a PR once I get some feedback from kivy-devs. $ rm -rf ~/.local/share/python-for-android/*.flag $ rm -rf ~/.local/share/python-for-android/build/bootstrap_builds/ $ rm -rf ~/.local/share/python-for-android/build/other_builds/ $ rm -rf ~/.local/share/python-for-android/build/python-installs/ $ rm -rf ~/.local/share/python-for-android/build/libs_collections/

Also if the dist name is the same for a Crystax and a Google build then the dist folder will also need to be removed because the dist will already contain the corrupted ".so" files and no build will occur. $ rm -rf ~/.local/share/python-for-android/dist/{dist-name}/

Alright now for the adb logcat output; but with out the full memory dump:

D/dalvikvm( 5140): Trying to load lib /data/app-lib/com.feralbytes.eztunes-1/libSDL2.so 0xadde42a8
E/dalvikvm( 5140): dlopen("/data/app-lib/com.feralbytes.eztunes-1/libSDL2.so") failed: dlopen failed: could not load library "libcrystax.so" needed by "libSDL2.so"; caused by library "libcrystax.so" not found
W/System.err( 5140): dlopen failed: could not load library "libcrystax.so" needed by "libSDL2.so"; caused by library "libcrystax.so" not found
D/dalvikvm( 5140): GC_FOR_ALLOC freed 1118K, 36% free 3159K/4876K, paused 14ms, total 14ms
I/dalvikvm-heap( 5140): Grow heap (frag case) to 3.839MB for 635812-byte allocation
D/dalvikvm( 5140): GC_FOR_ALLOC freed <1K, 23% free 3780K/4876K, paused 33ms, total 33ms
D/dalvikvm( 5140): GC_CONCURRENT freed <1K, 23% free 3787K/4876K, paused 7ms+12ms, total 53ms
W/dalvikvm( 5140): No implementation found for native Lorg/libsdl/app/SDLActivity;.nativeSetEnv:(Ljava/lang/String;Ljava/lang/String;)V
D/AndroidRuntime( 5140): Shutting down VM
W/dalvikvm( 5140): threadid=1: thread exiting with uncaught exception (group=0xadb56730)
E/AndroidRuntime( 5140): FATAL EXCEPTION: main
E/AndroidRuntime( 5140): java.lang.UnsatisfiedLinkError: Native method not found: org.libsdl.app.SDLActivity.nativeSetEnv:(Ljava/lang/String;Ljava/lang/String;)V
E/AndroidRuntime( 5140):     at org.libsdl.app.SDLActivity.nativeSetEnv(Native Method)
E/AndroidRuntime( 5140):     at org.kivy.android.PythonActivity$UnpackFilesTask.onPostExecute(PythonActivity.java:167)
E/AndroidRuntime( 5140):     at org.kivy.android.PythonActivity$UnpackFilesTask.onPostExecute(PythonActivity.java:114)
E/AndroidRuntime( 5140):     at android.os.AsyncTask.finish(AsyncTask.java:631)
E/AndroidRuntime( 5140):     at android.os.AsyncTask.access$600(AsyncTask.java:177)
E/AndroidRuntime( 5140):     at android.os.AsyncTask$InternalHandler.handleMessage(AsyncTask.java:644)
E/AndroidRuntime( 5140):     at android.os.Handler.dispatchMessage(Handler.java:99)
E/AndroidRuntime( 5140):     at android.os.Looper.loop(Looper.java:137)
E/AndroidRuntime( 5140):     at android.app.ActivityThread.main(ActivityThread.java:5103)
E/AndroidRuntime( 5140):     at java.lang.reflect.Method.invokeNative(Native Method)
E/AndroidRuntime( 5140):     at java.lang.reflect.Method.invoke(Method.java:525)
E/AndroidRuntime( 5140):     at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:737)
E/AndroidRuntime( 5140):     at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:553)
E/AndroidRuntime( 5140):     at dalvik.system.NativeStart.main(Native Method)
W/ActivityManager(  912):   Force finishing activity com.feralbytes.eztunes/org.kivy.android.PythonActivity

And this one shows that libpython is also affected:

D/dalvikvm( 5224): GC_CONCURRENT freed 424K, 16% free 4120K/4884K, paused 7ms+5ms, total 27ms
D/dalvikvm( 5224): Trying to load lib /data/app-lib/com.feralbytes.eztunes-1/libSDL2.so 0xadde76e8
D/dalvikvm( 5224): Added shared lib /data/app-lib/com.feralbytes.eztunes-1/libSDL2.so 0xadde76e8
D/dalvikvm( 5224): Trying to load lib /data/app-lib/com.feralbytes.eztunes-1/libSDL2_image.so 0xadde76e8
D/dalvikvm( 5224): Added shared lib /data/app-lib/com.feralbytes.eztunes-1/libSDL2_image.so 0xadde76e8
D/dalvikvm( 5224): No JNI_OnLoad found in /data/app-lib/com.feralbytes.eztunes-1/libSDL2_image.so 0xadde76e8, skipping init
D/dalvikvm( 5224): Trying to load lib /data/app-lib/com.feralbytes.eztunes-1/libSDL2_mixer.so 0xadde76e8
D/dalvikvm( 5224): Added shared lib /data/app-lib/com.feralbytes.eztunes-1/libSDL2_mixer.so 0xadde76e8
D/dalvikvm( 5224): No JNI_OnLoad found in /data/app-lib/com.feralbytes.eztunes-1/libSDL2_mixer.so 0xadde76e8, skipping init
D/dalvikvm( 5224): Trying to load lib /data/app-lib/com.feralbytes.eztunes-1/libSDL2_ttf.so 0xadde76e8
D/dalvikvm( 5224): Added shared lib /data/app-lib/com.feralbytes.eztunes-1/libSDL2_ttf.so 0xadde76e8
D/dalvikvm( 5224): No JNI_OnLoad found in /data/app-lib/com.feralbytes.eztunes-1/libSDL2_ttf.so 0xadde76e8, skipping init
D/dalvikvm( 5224): Trying to load lib /data/app-lib/com.feralbytes.eztunes-1/libpython2.7.so 0xadde76e8
E/dalvikvm( 5224): dlopen("/data/app-lib/com.feralbytes.eztunes-1/libpython2.7.so") failed: dlopen failed: could not load library "libcrystax.so" needed by "libpython2.7.so"; caused by library "libcrystax.so" not found
W/System.err( 5224): Couldn't load python3.5m from loader dalvik.system.PathClassLoader[DexPathList[[zip file "/data/app/com.feralbytes.eztunes-1.apk"],nativeLibraryDirectories=[/data/app-lib/com.feralbytes.eztunes-1, /system/lib]]]: findLibrary returned null

[FeralBytes]

Update the command must eliminate all of build so now passing to subprocess:

['rm', '-rf', os.path.join(os.path.expanduser('~'), '.local/share/python-for-android/build')]

[rockcastle]

My tts apk is working fine with crystax ndk https://github.com/rockcastle/Python-Kivy-Android-app

[yingshaoxo]

In fact, I wonder why author don't ant to builder IMG if Python for Android rely on it.

Because I got wrong when I try to build apk with it.

[rockcastle]

Check my buildozer spec file. I hope it helps https://github.com/rockcastle/Python-Kivy-Android-app?files=1

[yingshaoxo]

Thank you, I built successfully.

But is it normal to get a 19.6M .apk file?

On May 24, 2017 2:21 AM, "IsaBostan" notifications@github.com wrote:

Check my buildozer spec file. I hope it helps https://github.com/rockcastle/Python-Kivy-Android-app?files=1

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kivy/python-for-android/issues/992#issuecomment-303488367, or mute the thread https://github.com/notifications/unsubscribe-auth/AQZPraeL6hTH6TO1e3fZ9rBcpbDnL9EKks5r8yOdgaJpZM4LsuMc .

[rockcastle]

I have similar sizes for my apps. import specific libs to make it small size.

[inclement]

CrystaX-built APKs are about 9MB larger than they need to be due to a bug in what is included. You can make them much smaller by locating the dist directory and deleting the usr directory, which I think is in something like dist_dir/crystax_python/site-packages. I'll check the exact path when I next make a python3 APK.

[yingshaoxo]

And there has another problem, Buildozer currently does not support built apk with python3crystax.

If you import something like flask, it'll get wrong due to python2.

https://github.com/kivy/python-for-android/issues/1041

2017-05-25 6:17 GMT+08:00 Alexander Taylor notifications@github.com:

CrystaX-built APKs are about 9MB larger than they need to be due to a bug in what is included. You can make them much smaller by locating the dist directory and deleting the usr directory, which I think is in something like dist_dir/crystax_python/site-packages. I'll check the exact path when I next make a python3 APK.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kivy/python-for-android/issues/992#issuecomment-303867827, or mute the thread https://github.com/notifications/unsubscribe-auth/AQZPrdbZwzsCkG8ulU2W5P0PjWh9RyqRks5r9KyIgaJpZM4LsuMc .

[rockcastle]

Hi, "dist_dir/crystax_python/site-packages" I have checked to make it smaller size but it did not worked for me. Is that the place in below? .buildozer/android/platform/build/dists/app_name/crystax_python/crystax_python/site-packages/

[AndreMiras]

CrystaX-built APKs are about 9MB larger than they need to be due to a bug in what is included. You can make them much smaller by locating the dist directory and deleting the usr directory, which I think is in something like dist_dir/crystax_python/site-packages. I'll check the exact path when I next make a python3 APK.

@inclement that's super interesting, I would do anything to have my APKs 9M smaller! Until we make a built-in fix, is this something that can be workarounded with a blacklist.txt file? Do you have more info, I'm interested in looking at it. Can we open a dedicated ticket for this topic?

Edit: following up our discussion on IRC, it does not seem to be an issue anymore. For record the problem was ...crystax_python/site-packages/ contained a usr/ directory with a lot of stuff inside. I've checked on my build and this directory was pretty light. So indeed not an issue anymore.

[homdx]

Hello. I managed to save files when they were successfully collected and you can download them from my page or from the docker: https://github.com/homdx/KivyMD/releases

[ghost]

@homdx thanks, that will be useful for anyone dropping by via google :heart:

@malverick thanks for the report! I have been not involved in this discussion/ticket, but from what I can see this is mostly around CrystaX which is no longer needed: there is a regular python3 recipe now which uses the regular Google NDK! Also, the discussion appears to have gone astray from the initial issue.

I'll therefore close this for now, but feel absolutely free to poke me if there is still a problem with the particular issue you initially reported and I'll reopen - so far it's to my knowledge still undecided if/when CrystaX will be fully dropped, after all