Google requiring API Target 26 in Aug/Nov 2018

gorgonaut04 commented 6 years ago

Just wanted to put it on the devs’ radar. We’ve been limited to 19 for the last couple years, afaik, but kivy will need to support targeting 26 for new apps starting in August, and for app updates starting in November: https://android-developers.googleblog.com/2017/12/improving-app-security-and-performance.html?m=1

If there’s currently a straightforward way to target 21 or higher then I’ll stand corrected and be very interested in hearing how.

Zen-CODE commented 5 years ago

Frikkin yay! Got Python3 building with the CrystaxNDK, API 27. It required me to copy the "platform/android-27" folder from a newer SDK into the Crystax NDK "platform/android-27" folder and that works. Lots of trials and tribulations along the way, but such is the journey of learning.

But to the point. I will now finalise, clean and compact the "Kivy Complete VM", upload it and announce it's availability on the forums. It will be able build Python2 or Python3 APK's with API 19 or 27. I've removed support for other API's as they are not really valuable and quite size-able on disk.

@inclement. Regarding making this work easily with buildozer as is, I think I will need to start from scratch now that I have a better understanding of how stuff fits together and what the end-system looks like. Its (seems) good news we can still use the android 23 SDK as the "android" executable functionality seems to move to "bin/sdkmanager" with later SDK's. That can be addressed in time....

Thanks again for all the great work you have+are doing. Respect.

inclement commented 5 years ago

It required me to copy the "platform/android-27" folder from a newer SDK into the Crystax NDK "platform/android-27" folder and that works.

Nice, a convenient hack!

Its (seems) good news we can still use the android 23 SDK as the "android" executable functionality seems to move to "bin/sdkmanager" with later SDK's.

python-for-android should already work with this, if it doesn't it's a bug rather than a totally missing feature.

edsonitk commented 5 years ago

Hi Zen-Code, How did you build apk using ndk r16b and API 27 for python 2? I can generate the apk but there is always an error in the execution: SDL Error dlopen failed: can not locate simbol "ftello64" referenced by "libpython2.7.so"

Zen-CODE commented 5 years ago

Hi

I am about to make a post announcing the release to a VM for building those APKs and with instructions that might help. I will let you know when that is done.

Cheers

On Tue, 17 Jul 2018, 00:19 edsonitk, notifications@github.com wrote:

Hi Zen-Code, How did you build apk using ndk r16b and API 27 for python 2? I can generate the apk but there is always an error in the execution: SDL Error dlopen failed: can not locate simbol "ftello64" referenced by " libpython2.7.so"

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kivy/python-for-android/issues/1219#issuecomment-405399268, or mute the thread https://github.com/notifications/unsubscribe-auth/ADYDK14i9JRxvMn_wWYJHZjhRfNCPRVtks5uHRGKgaJpZM4RZ5U5 .

Zen-CODE commented 5 years ago

Really don't see how that change address the issue of supporting API 26+

AndreMiras commented 5 years ago

Sorry my bad, because of that error BCMDisplayException in the conversation, I did a quick search to see if it was addressing somewhat duplicated issues. Apparently not that one. Thanks for reopening :smile:

ApplySci commented 5 years ago

@Zen-CODE did you get chance to make a post announcing the release to a VM for building those APKs and with instructions that might help? If so, please can you link to that post here?

Zen-CODE commented 5 years ago

It turns out that although the build succeeds, the upload to the store fails and says the APK is still using API 19. I'm looking into that now and will post when I know it is working.

Thanks

On Thu, Aug 16, 2018 at 9:10 AM ApplySci notifications@github.com wrote:

@Zen-CODE https://github.com/Zen-CODE did you get chance to make a post announcing the release to a VM for building those APKs and with instructions that might help? If so, please can you link to that post here?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kivy/python-for-android/issues/1219#issuecomment-413448278, or mute the thread https://github.com/notifications/unsubscribe-auth/ADYDK5atBeqW7uTaqXeZVFDSKtpksPXxks5uRRrTgaJpZM4RZ5U5 .

-- Richard Larkin

The Zen of Kivy: Open Source, cross platform, full stack. Mobile and desktop. http://kivy.org/

http://www.camiweb.com/zenkey

ApplySci commented 5 years ago

huh. how frustrating for you. Thanks for the update, and good luck with getting this sorted!

edsonitk commented 5 years ago

By the various tests that I did, it seems that the maximum api accepts is 22. Regardless of the SDK and NDK version. Above api 22 creates apk but error occurs on app load: symbol ftello64 referenced by libpython2.7.so

Setup: kivy == 1.10.1 or master, python 2.7, cython == 0.21 or greater, buildozer 0.34 or Dev, p4a == master or stable. NDK 16b, SDK (20,23,24,25,27)

rkhimura commented 5 years ago

I trust in God that you can, I really like kivy

SomberNight commented 5 years ago

What Google actually checks is the targetSdkVersion which is basically independent of the compileSdkVersion, and you can just set it to anything. It's specified here: https://github.com/kivy/python-for-android/blob/275d2360840b73cb968d2f8a9e7c833c9d2ca4fd/pythonforandroid/bootstraps/sdl2/build/templates/build.tmpl.gradle#L27

However, the caveat is that if you set it to higher than compileSdkVersion, then the OS will expect your APK to be able to handle things that you might not even have the API handles for. For example, runtime permissions were introduced in sdk 23. If you set targetSdkVersion to >=23, then Android will expect it to handle runtime permission dialogs, but if you compiled the apk with an older SDK version, then you won't have the API calls to handle those dialogs.

atnccetti commented 5 years ago

Sorry for my weak English .... In the original Buildozer Vm: (https://kivy.org/#download = buildozer -v 0.32 and python for android -v .04), the android api 26 and 27 are only accepted for release mode compilation, but for debugging will fail, the store accepts the apk, the bad thing is that I can not use sdl2, and it does not seem to download the api in fact, strange !! ..., thank you!.

rkhimura commented 5 years ago

Thank you, Zen-CODE, inclement, 40four, AndreMiras, SomberNight, for your time and effort.

I made my application with buildozer with the default configuration (api 19), I do not know how to change the targetSdk for my application to api 27?

SomberNight has already clarified that I can change the targetSdkVersion, but I do not know where or how to change that parameter. With Python for Android directly or with buildozer?

But I think it's better to wait for Zen-CODE to announce the release to a VM to build those APKs with instructions.

Zen-CODE commented 5 years ago

@inclement. Frustrating. IRC seems blocked, so will post here. Looking at the API 27 issue. When the app runs, we get an 'pythonutil: Library loading error: dlopen failed: cannot locate symbol "__aeabi_memclr8" referenced by "/data/app/org.test.myapp-1/lib/arm/libSDL2.so"' error.

Looking around, it seems these symbols were moved from LIBC to LIBC_N in the API 23 change.

https://android.googlesource.com/platform/bionic/+/6d142bcf34ffd49efaf4285bb2af63a1636706f9^!/

I find this change in 'android-nkd_r16b/platforms/android-23/arch-arm/usr/lib/lic.a' file in the NDK. But I'm not sure where we change this to make lbSDL2.so work?

inclement commented 5 years ago

About IRC, you may need to register your username to post at the moment due to an influx of spam. I thought we'd turned this off though, and have raised a query about it.

matham commented 5 years ago

@Zen-CODE sorry about the IRC block. There was a ton of spam for a few of weeks so I had to block unregistered users from posting. Looks like it's finally over now, though, so I unblocked and anyone can post.

inclement commented 5 years ago

@Zen-CODE My Python 3 PR has some notes about this. In particular, introducing APP_PLATFORM set to an appropriate value is the fix for your __aeabi_memclr8__ issue, which I also saw along the way. Essentially, it means that SDL2 can be built against an older platform version even though the Java API target is much newer. We should have been setting this anyway, if it isn't set explicitly then as far as I can tell the NDK will pick a probably-wrong default value.

inclement commented 5 years ago

Anyone trying to target API 26 may like to try the set_app_platform_19 branch - I'm not certain if it will work, but it can't hurt.

tito commented 5 years ago

Could APP_PLATFORM := android-19 be changed with APP_PLATFORM := $(APP_PLATFORM) and set it from the buid env?

inclement commented 5 years ago

Yep, I intend to make a proper PR for this if nobody else does first. I just made the branch to make it clear to people what the change is, so they can understand it and test easily if they want to do so immediately.

Edit: Specifically, I think the APP_PLATFORM should by default be the same as the minSdk: you wouldn't normally want it to be higher, and it's unlikely to need to be lower unless your minSdk is very new. However I would make it fully configurable.

Zen-CODE commented 5 years ago

@inclement. Progress. The app launches but on the device gives an "SDL Error" popup.

Error: dlopen failed: cannot locate symbol "ftello64" referenced by "/data/app/org.test.myapp-2/lib/arm/libpython2.7.so"...

Looking through the logcat output, this happens right after an error trying to find "libpython3.5m.so", but I doubt that's important now. This missing symbol seems to cause the crash...

inclement commented 5 years ago

What targetSDK, compileSDK, NDK_PLATFORM and minSDK are you using?

Zen-CODE commented 5 years ago

The spec file only contains "android.api = 27, android.sdk=23" and the ndk path is to the the extracted "android-ndk-r16b"

inclement commented 5 years ago

And you're using my set_app_platform_19 branch, or otherwise setting the APP_PLATFORM some other way?

Zen-CODE commented 5 years ago

Yeah, using that branch. And starting without any previous .buildozer folder in the app directory. And device is android 6 (API level 23 I believe)

Zen-CODE commented 5 years ago

Update. The "set_app_platform_19" branch still works with android.api = 19. The log also shows the python3.5m error but then carries on loading the other libs as normal. ...

Zen-CODE commented 5 years ago

In case it helps, the below mentions a commit "exposing" ftello64.

https://www.androidpolice.com/android_aosp_changelogs/android-m-preview-2-to-android-n-preview-1-AOSP-changelog.html

nkiiiiid commented 5 years ago

hi, i have working on android package with api 27 ,i add APP_PLATFORM := android-12 to pythonforandroid/bootstraps/sdl2/build/jni/Application.mk then package, got the same error as @Zen-CODE report
Error: dlopen failed: cannot locate symbol "ftello64" referenced by "/data/app/org.test.myapp-2/lib/arm/libpython2.7.so"...

vm setup:
ubuntu 16.04 python2.7 kivy 1.10.1 cython0.25.2
p4a==master sdk api level=27 ndk r16b build tool 27.0.3

any suggestion？ i have deal with it almost one week and feel so despair

linuxrootok commented 5 years ago

I think that the SDK API 26 version can not be packaged,that's horrible. Hope that it will be fixed sooner,I love KIVY project.

clevermindgames commented 5 years ago

I was able to build my app with gradle (from buildozer) using the following:

android.api = 27 android.minapi = 19

Importantly, when uploaded to Google Play there is no message saying I am targeting old API. I have build tools version 28.0.2, but I was also successfully building and uploading to Google Play using 26.0.1 Admob ads seems to be working fine too using default bootstrap (I am using monkeypatched Kivmob version).

Using p4a cloned about a week ago and kivy=master. Latest version of buildozer.

There are a couple of issues I did not manage to resolve yet: 1) At the end of the process I get the following output: To run dex in process, the Gradle daemon needs a larger heap. [DEBUG]: It currently has 1024 MB. [DEBUG]: For faster builds, increase the maximum heap size for the Gradle daemon to at least 1536 MB. To run dex in process, the Gradle daemon needs a larger heap. [DEBUG]: It currently has 1024 MB. [DEBUG]: For faster builds, increase the maximum heap size for the Gradle daemon to at least 1536 MB. To do this set org.gradle.jvmargs=-Xmx1536M in the project gradle.properties.

Unfortunately, even adding this line to all gradle.properties files in .buildozer folder and my p4a directory does not result in any change to this message. This is basically means there are very few SDK I can add to my App (at least this is how I read it - I might be wrong)

2) Due to my app targetting API=27, all permissions required have to be requested dynamically when user runs the app for the first time. User should have a dialog to either allow or deny each permission like "write external storage". Unfortunately, there is no handle for this in kivy yet, so it is a major problem. The only work around I see is to write a message to the user and explain how to grant permissions manually. Terrible user experience.

Zen-CODE commented 5 years ago

@clevermindgames. Using the same setting, I get the "ftello64" error. But I have build tools 23, 25 + 28 installed. How do you specify which one to use? Also, which SDK version are you using?

re: point 2. Is there a way to do that even in Java? I would have thought on a per-request basis is how later versions of android handle the issue by default....

mkg20001 commented 5 years ago

Just fyi I'm currently successfully publishing https://github.com/HelloZeroNet/ZeroNet-kivy for SDK 26 with the following config: https://github.com/HelloZeroNet/ZeroNet-kivy/blob/master/buildozer.spec#L81-L91 Note that I'm using a custom tool https://github.com/HelloZeroNet/ZeroNet-kivy/tree/master/buildozer-android-downloader to download the newer SDKs / NDKs and my own fork of p4a with some PRs merged in https://github.com/mkg20001/python-for-android/

nkiiiiid commented 5 years ago

@mkg20001 zeronet alpha#10 crash on andriod 6.0 (samsung) , work well on Android8.0/8.1. it seems package with api 26 can't compatible with Android 6.0 and below.

mkg20001 commented 5 years ago

Didn't notice this myself because I use android 8.1. Is there any fix available?

海森堡测不准面包 notifications@github.com schrieb am Do., 20. Sep. 2018, 07:57:

@mkg20001 https://github.com/mkg20001 zeronet alpha#10 crash on andriod 6.0 (samsung) , work well on Android8.0/8.1. it seems package with api 26 can't compatible with Android 6.0 and below.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kivy/python-for-android/issues/1219#issuecomment-423049396, or mute the thread https://github.com/notifications/unsubscribe-auth/AHYHaUi6ZePshs17_fUnYiRpBX6LKzOFks5ucy5GgaJpZM4RZ5U5 .

clevermindgames commented 5 years ago

@clevermindgames. Using the same setting, I get the "ftello64" error. But I have build tools 23, 25 + 28 installed. How do you specify which one to use? Also, which SDK version are you using?

From Android SDK manager: Android SDK Tools: 25.2.5 Android SDK Platform Tools: 28.0.1 Android SDK Build-tools: 28.0.2, 26.0.1, 23.0.1, 19.1 SDK Platform: 28, 27, 26, 19 (targeting 27 in buildozer, 28 and 26 did not work) Extras -> Android Support Repository: 47 Have serious doubts I really need them all, just did not bother to remove those which did not work out It always uses the highest version available, so if I need to use build-tools 26.x over 28.x I just open SDK manager and uninstall 28.x

I also did not work with android.minapi lower than 19. Might be due to some poor back compatability

re: point 2. Is there a way to do that even in Java? I would have thought on a per-request basis is how later versions of android handle the issue by default....

There was no solution 4 months ago, but it looks to be feasible: https://www.reddit.com/r/kivy/comments/8kl10p/is_there_a_way_to_implement_runtime_permissions/

Zen-CODE commented 5 years ago

@mkg20001. Thanks a lot for the info. Will look into that. What versions of Android have you tested on?

@clevermindgames. Okay, I'm also experimenting with using Android studio to manage the SDK's and API's. I've now built using API 27 + SDK 28 + Platform tools 28 but still hitting the 'ftello64' error when running the app....Also using 19 as minapi.

mkg20001 commented 5 years ago

@Zen-CODE Currently only 8.1. From user feedback it seems like the app crashes on any device below API 26 but I have no idea what could be causing this. Example: https://github.com/HelloZeroNet/ZeroNet/issues/1624 but there are many more of these (this is the most common crash)

Malakanorth commented 5 years ago

With the same settings as @Zen-CODE , the same error when I launch my APK on android 6 Error: dlopen failed: cannot locate symbol "ftello64" referenced by "/data/app/org.test.myapp-2/lib/arm/libpython2.7.so"

When I launch the same APK on android 7 the device gives this error System.err: dlopen failed: cannot locate symbol "ctermid" referenced by "/data/app/com.cassetete-1/lib/arm/libpython2.7.so"

Malakanorth commented 5 years ago

OK my app build with NDK r16b and API 27. No more errors related to ftello64.

In first build with API 19, when it's good remove your dist from .buildozer/android/platforms/build/dists and re-build with API 27.

I have upload my APK to Play Store without errors related to API used.

So now when I launch my APK I have an error 09-24 00:29:00.352 26402 26448 I python : File "/home/malak/Documents/casse_tete/.buildozer/android/platform/build/dists/labyrinthecassetete/private/lib/python2.7/site-packages/kivy/app.py", line 780, in user_data_dir 09-24 00:29:00.353 26402 26448 I python : OSError: [Errno 13] Permission denied: '/sdcard/cassetete' This error is related to WRITE permission, I work around this now.

If I remove the lines who used user_data_dir, it works!!

Tested on Android 6.

clevermindgames commented 5 years ago

Well done, I had no idea API 19 was required, never seen dlopen failed error myself. Looks like I got it (API 19 -> API 27 ) by accident since my previous builds were targetting API 19 anyway.

Spent quite a lot of time trying to find a way to store data in internal (private) storage for the app so it is not getting deleted every time user exits the app. Here is the solution:

from kivy.utils import platform
if platform.lower() == "android":
        from jnius import autoclass, cast
        PythonActivity = autoclass('org.kivy.android.PythonActivity')
        Environment = autoclass('android.os.Environment')
        context = cast('android.content.Context', PythonActivity.mActivity)
        private_storage = context.getExternalFilesDir(Environment.getDataDirectory().getAbsolutePath()).getAbsolutePath();

inclement commented 5 years ago

For some perspective on why the API 19 build works: I think you are effectively achieving the same thing as setting APP_PLATFORM, such that the NDK libraries are built for api 19 even though the APK target API is higher. This is actually a kind of bug in p4a, it should not use the same build in this case.

Zen-CODE commented 5 years ago

@StacyMAI. Thanks a lot for sharing that, but I am confused. I've used the same NDK and API, and the same "build for 19 then 27" approach but got reports that the APK was still refused by google. The only difference is that you clear the "dists" folder before rebuilding. I can confirm it solves the ftello error and runs on my phone, but I will reserve my explosions of gratitude and wonder when the store accepts the APK.

Unfortunately, it will take me a little while to test that. Our current app only builds with the old toolchain, so I need to solve those issues before I can build a new APK and upload. I will post back when I can confirm that it is accepted. Until then, I would encourage others to post their findings when using this process and uploading.

rnixx commented 5 years ago

Hi all,

I currently try to update my app as well. Following is defined in my buildozer.spec:

android.api = 27
android.minapi = 19
android.sdk = 23
android.ndk = 16b

I deleted all buildozer stuff and started completly from scratch. First I had to download NDK r16b manually, because the automatic download failed and define the NDK location in the spec file like so:

android.ndk_path = ~/.buildozer/android/android-ndk-r16b

Next i applied the changes from https://github.com/kivy/python-for-android/compare/set_app_platform_19 on my p4a fork (after merging changes from master).

Then I needed to modify ~/.buildozer/android/platform/android-sdk-23/tools/ant/build.xml and set java.source and java.target to 1.7.

I ran into System.err: dlopen failed: cannot locate symbol "..." referenced by ... errors with ctermid and setpwent for which i added patches disabling the code using them here https://github.com/rnixx/python-for-android/compare/master...rnixx:ctermid_getpwall_patches

Looking into the related code from python 2.7.2, these patches might be obsolete if I find a way for setting values at cmmi configure time, when pyconfig.h gets generated from pyconfig.h.in. It looks like default values are taken from how system or hostpython was compiled. Unsetting HAVE_CTERMID and HAVE_GETPWENT will probably do the trick, however, i have no idea how to achieve this.

Finally the apk builds, but now i get random ImportError: dlopen failed: file offset for the library "/data/user/0/.../files/app/lib/python2.7/site-packages/kivy/_...so" >= file size: 0 >= 0 for several modules. _properties.so, _event.so and android.runnable are such candidates. For each build it's always the same missing module, but when deleting the build and rebuilding the apk, some other stuff is missing. I have no idea what's going on here and would be really glad if someone point me in the right direction.

Thanks, Robert

inclement commented 5 years ago

@Zen-CODE Another aspect of p4a's buggy target api handling is that dists are fixed to a specific target api once created. It's written to one of their config files. I only realised this recently, I realise it must have caused quite some confusion.

inclement commented 5 years ago

@rnixx The problem here is probably not that they have file size 0 (which is true and actually okay), but that libpymodules.so has not been included or loaded as it should be. That file should provide the symbols you need. When it hasn't been loaded the error message is misleading because it doesn't know about the slightly weird symbol redistribution p4a does.

rnixx commented 5 years ago

@inclement for some reason the libs are now included properly, maybe because i changed the requirements recipe order. don't know. further android recipe was missing. now - hopefully the final issue - i get a permission denied error to the SD card, even if READ_EXTERNAL_STORAGE and WRITE_EXTERNAL_STORAGE permissions are granted. do you have an idea what's missing?

OptimusGREEN commented 5 years ago

@StacyMAI. Thanks a lot for sharing that, but I am confused. I've used the same NDK and API, and the same "build for 19 then 27" approach but got reports that the APK was still refused by google. The only difference is that you clear the "dists" folder before rebuilding. I can confirm it solves the ftello error and runs on my phone, but I will reserve my explosions of gratitude and wonder when the store accepts the APK.

Unfortunately, it will take me a little while to test that. Our current app only builds with the old toolchain, so I need to solve those issues before I can build a new APK and upload. I will post back when I can confirm that it is accepted. Until then, I would encourage others to post their findings when using this process and uploading.

I tested building with 19 and then removing the dists folder as suggested by @StacyMAI but upon opening my app I get: IOError: [Errno 13] Permission denied: '/storage/emulated/0/OptiLEAVE/saved_data/currentuser.xml'

This is just reading an xml which is previously saved data.

The app has read and write permissions in the spec file.

I can also confirm that in using this method, play console does see the API as 27.

inclement commented 5 years ago

Are these various read and write issues just caused by the fact that permissions changed in Android 26 or so: now you have to ask for them with the java api, I haven't tried this.

OptimusGREEN commented 5 years ago

I would have thought that would be something that android handled itself.

kivy / python-for-android

Google requiring API Target 26 in Aug/Nov 2018 #1219

-- Richard Larkin