Unable to open a zip on Android: zipfile.BadZipfile: File is not a zip file

[Zen-CODE]

Versions

• Python: 2.7.2
• OS: ubuntu 16.04, android 6
• Kivy: 1.10.0
• Cython: 0.25.2

Description

This is my setup.

cython 0.25.2 ubuntu 16.04 android ndk r16b android sdk 28 api=27 python-for-android - using branch set_app_platform_19

the requirements contains pyjnius, and other calls to it works e.g. starting the service.

The app runs and imports work fine. The service starts, but it fails on the first import. The import is valid (from the root of the app folder, not the service folder), but it crashes with a jnius.so error. Even if the error is trapped, it just crashes without going into the exception clause.

The error is logged here.

https://gist.github.com/Zen-CODE/0a11a5881bb80caeeee17b8cb5a2d48e

buildozer.spec

Command:

buildozer android debug deploy run

Spec file:

[app]

# (str) Title of your application
title = CAMI-Apps

# (str) Package name
package.name = camiweb.com

# (str) Package domain (needed for android/ios packaging)
package.domain = speedtest.maths

# (str) Source code where the main.py live
source.dir = .

# (list) Source files to include (let empty to include all the files)
# source.include_exts = py,png,jpg,kv,atlas,ini

# (list) Source files to exclude (let empty to not exclude anything)
#source.exclude_exts = spec

# (list) List of directory to exclude (let empty to not exclude anything)
#source.exclude_dirs = working

# (list) List of exclusions using pattern matching
#source.exclude_patterns = license,images/*/*.jpg

# (str) Application versioning (method 1)
version = 2.1.10

# (str) Application versioning (method 2)
# version.regex = __version__ = ['"](.*)['"]O
# version.filename = %(source.dir)s/main.py

# (list) Application requirements
# comma seperated e.g. requirements = sqlite3,kivy
# requirements = python, sqlite3, kivy==1.10.0, openssl
# For new toolchain
requirements = sqlite3, kivy==1.10.0, openssl, pyjnius, android

# (str) Custom source folders for requirements
# Sets custom source for any requirements with recipes
# requirements.source.kivy = ../../kivy

# (list) Garden requirements
#garden_requirements =

# (str) Presplash of the application
presplash.filename = working/splash/presplash.jpg

# (str) Icon of the application
icon.filename = %(source.dir)s/icon.png

# (str) Supported orientation (one of landscape, portrait or all)
orientation = landscape

# (list) List of service to declare
#services = NAME:ENTRYPOINT_TO_PY,NAME2:ENTRYPOINT2_TO_PY
services = camiapps:service/main.py

#
# OSX Specific
#

#
# author = © Copyright Info

#
# Android specific
#

# (bool) Indicate if the application should be fullscreen or not
fullscreen = 1

# (list) Permissions
android.permissions = INTERNET

# (int) Android API to use
android.api = 27

# (int) Minimum API required
android.minapi = 19

# (int) Android SDK version to use
# android.sdk = 20
android.sdk = 28

# (str) Android NDK version to use
#android.ndk = 9c

# (bool) Use --private data storage (True) or --dir public storage (False)
#android.private_storage = True

# (str) Android NDK directory (if empty, it will be automatically downloaded.)
android.ndk_path = /home/kivy/Android/android-ndk-r16b/

# (str) Android SDK directory (if empty, it will be automatically downloaded.)
#android.sdk_path =

# (str) ANT directory (if empty, it will be automatically downloaded.)
#android.ant_path =

# (str) python-for-android git clone directory (if empty, it will be automatically cloned from github)
#android.p4a_dir =
p4a.source_dir = /home/kivy/Repos/python-for-android/

# (list) python-for-android whitelist
#android.p4a_whitelist =

# (str) Android entry point, default is ok for Kivy-based app
#android.entrypoint = org.renpy.android.PythonActivity

# (list) List of Java .jar files to add to the libs so that pyjnius can access
# their classes. Don't add jars that you do not need, since extra jars can slow
# down the build process. Allows wildcards matching, for example:
# OUYA-ODK/libs/*.jar
#android.add_jars = foo.jar,bar.jar,path/to/more/*.jar

# (list) List of Java files to add to the android project (can be java or a
# directory containing the files)
#android.add_src =
#android.add_src = java/com

# (str) python-for-android branch to use, if not master, useful to try
# not yet merged features.
#android.branch = master

# (str) OUYA Console category. Should be one of GAME or APP
# If you leave this blank, OUYA support will not be enabled
#android.ouya.category = GAME

# (str) Filename of OUYA Console icon. It must be a 732x412 png image.
#android.ouya.icon.filename = %(source.dir)s/data/ouya_icon.png

# (str) XML file to include as an intent filters in <activity> tag
#android.manifest.intent_filters =

# (list) Android additionnal libraries to copy into libs/armeabi
#android.add_libs_armeabi = libs/android/*.so
#android.add_libs_armeabi_v7a = libs/android-v7/*.so
#android.add_libs_x86 = libs/android-x86/*.so
#android.add_libs_mips = libs/android-mips/*.so

# (bool) Indicate whether the screen should stay on
# Don't forget to add the WAKE_LOCK permission if you set this to True
#android.wakelock = False

# (list) Android application meta-data to set (key=value format)
#android.meta_data =

# (list) Android library project to add (will be added in the
# project.properties automatically.)
#android.library_references =

# (str) Android logcat filters to use
#android.logcat_filters = *:S python:D

# (bool) Copy library instead of making a libpymodules.so
#android.copy_libs = 1

#
# iOS specific
#

# (str) Name of the certificate to use for signing the debug version
# Get a list of available identities: buildozer ios list_identities
#ios.codesign.debug = "iPhone Developer: <lastname> <firstname> (<hexstring>)"

# (str) Name of the certificate to use for signing the release version
#ios.codesign.release = %(ios.codesign.debug)s

[buildozer]

# (int) Log level (0 = error only, 1 = info, 2 = debug (with command output))
log_level = 2

# (int) Display warning if buildozer is run as root (0 = False, 1 = True)
warn_on_root = 1

#    -----------------------------------------------------------------------------
#    List as sections
#
#    You can define all the "list" as [section:key].
#    Each line will be considered as a option to the list.
#    Let's take [app] / source.exclude_patterns.
#    Instead of doing:
#
#[app]
#source.exclude_patterns = license,data/audio/*.wav,data/images/original/*
#
#    This can be translated into:
#
#[app:source.exclude_patterns]
#license
#data/audio/*.wav
#data/images/original/*
#

#    -----------------------------------------------------------------------------
#    Profiles
#
#    You can extend section / key with a profile
#    For example, you want to deploy a demo version of your application without
#    HD content. You could first change the title to add "(demo)" in the name
#    and extend the excluded directories to remove the HD content.
#
#[app@demo]
#title = My Application (demo)
#
#[app:source.exclude_patterns@demo]
#images/hd/*
#
#    Then, invoke the command line with the "demo" profile:
#
#buildozer --profile demo android debug

Logs

https://gist.github.com/Zen-CODE/0a11a5881bb80caeeee17b8cb5a2d48e

[Zen-CODE]

It seems to be that this issue is related to variable values not being shared by the main app and the service. To elaborate.

There is a dictionary in a file "helpers/localos.py". This dictionary is used as a singleton and populated on the first call to some of the module functions. From there on, the module functions return values loaded from the stored dictonary. This means that we only call the autoclass function once.

Using the old toolchain, the dictionary persisted between the app and the service.

Using the new toolchain, the dictionary filled by the app is empty for the service. The service then tries to populate it and crashes on the 2nd call of:

activity = autoclass('org.kivy.android.PythonActivity')

Is there any known reason when this might be happening? The loss of state between the app and the service is cetainly different behaviour.

Dictionary persists between app and service
    buildozer command: buidlozer android_old
    python-for-android branch: old_toolchain
    android sdk: 23
    android ndk: 9c

Dictionary does not persist between app and service
    buildozer command: buidlozer android
    python-for-android branch: set_app_platform_19
    android sdk: 28
    android ndk: r16b

Would it help if I prepared a minimal example demonstrating the issue? Or is there a known cause?

[Zen-CODE]

Could this be due to the changed "Class-loading behavior" described in the changes here?

https://developer.android.com/about/versions/oreo/android-8.0-changes

[Zen-CODE]

Adding code to avoid the above issues, it seems that using the new toolchain, the zipfile module cannot open a valid zipfile. It throws the error

    File is not a zip file

But I have verified it is by opening it with another tool. And the same file can be opened by zipfile using the android_old toolchain....

[AndreMiras]

Thanks for investigating :+1:

Would it help if I prepared a minimal example demonstrating the issue? Or is there a known cause?

Yes please do. I'm not sure I'm following, but know that at least on the new toolchain the service and the main app are two different process, so the memory is not shared. See https://github.com/kivy/python-for-android/commit/ee58deeea9ef44d2d9a11a6b2314b626e06fde89#diff-419ff95ffcc5c8bb0c2b40a76a023bb2R12

[Zen-CODE]

It turns out that neither the service not the app itself can open a normal zipfile. The exact same zip was open-able using the android_old toolchain and the app opens the zip no problem. It only fails when packaged as an APK. The error log is here:

https://gist.github.com/Zen-CODE/867c8585d29e54a638063f2964366fdc

The minimal project that demonstrates the issue is here.

https://github.com/Zen-CODE/kivybits/tree/master/Examples/Services

Although the app has storage permissions, I suspect that it has do with the new protocol of requesting runtime permissions, and the 'zipfile.BadZipfile: File is not a zip file' message really means 'permission denied'. It will test this by seeing if I can open the file as a binary and read bits from it....

Update. Nope. The file can be read as a binary file using open(my_zip, 'rb') no problem. So it's not a permissions problem. So why is the zipfile module failing to open it?

[rasmuskreiner]

I had the exact same issue, and decided to do a work around with pyjnius. See this java-class: https://pastebin.com/pxJnEiR7

[OptimusGREEN]

I had the exact same issue, and decided to do a work around with pyjnius. See this java-class: https://pastebin.com/pxJnEiR7

This link isn't working but yes I have the same issue. tar file doesn't extract either.

[rasmuskreiner]

I had the exact same issue, and decided to do a work around with pyjnius. See this java-class: https://pastebin.com/pxJnEiR7

This link isn't working but yes I have the same issue. tar file doesn't extract either.

Link should be working now.... BTW usage is as follows:

unzip = autoclass("unzipFile")
unzipper = unzip()
unzipper.extractFile(filePathToZipFile, filePathToDestinationFolder)

[Zen-CODE]

@rasmuskreiner. Thanks. I'll keep that in mind. But what worries me is that other things are also failing. E.g. If I try and autoclass in the service, it also crashes. So there is somethings more sinister afoot methinks...

[Zen-CODE]

@rasmuskreiner. I'm struggling to get the java class detected. How do you include you java class file in your buildozer.spec file? Using "android.add_src" or "android.add_jars"? Do you have to do anything extra to register this class on android? e.g. set the CLASSPATH? Thanks

[Zen-CODE]

@OptimusGREEN. Were you trying to extract on a background thread by any chance? Looks like that might be the cause of my issues. Still investigating....

[OptimusGREEN]

Hi, No I don't think I was using threading. 

Sent from my Samsung Galaxy smartphone. -------- Original message --------From: Richard Larkin notifications@github.com Date: 06/01/2019 13:19 (GMT+00:00) To: kivy/python-for-android python-for-android@noreply.github.com Cc: OptimusGREEN johnmcbridegreen@gmail.com, Mention mention@noreply.github.com Subject: Re: [kivy/python-for-android] Unable to open a zip on Android: zipfile.BadZipfile: File is not a zip file (#1388) @OptimusGREEN. Were you trying to extract on a background thread by any chance? Looks like that might be the cause of my issues. Still investigating....

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread. {"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/kivy/python-for-android","title":"kivy/python-for-android","subtitle":"GitHub repository","main_image_url":"https://github.githubassets.com/images/email/message_cards/header.png","avatar_image_url":"https://github.githubassets.com/images/email/message_cards/avatar.png","action":{"name":"Open in GitHub","url":"https://github.com/kivy/python-for-android"}},"updates":{"snippets":[{"icon":"PERSON","message":"@Zen-CODE in #1388: @OptimusGREEN. Were you trying to extract on a background thread by any chance? Looks like that might be the cause of my issues. Still investigating...."}],"action":{"name":"View Issue","url":"https://github.com/kivy/python-for-android/issues/1388#issuecomment-451741092"}}} [ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "https://github.com/kivy/python-for-android/issues/1388#issuecomment-451741092", "url": "https://github.com/kivy/python-for-android/issues/1388#issuecomment-451741092", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

[rasmuskreiner]

Sorry the late response. I’m on paternity leave hence the slow reply rate…

As I recall I was also struggling with this part. Ultimately I ended up putting the java files in the root directory, and adding them to the “android.add_src". Eg. android.add_src = unzipFile.java.

Den 30. dec. 2018 kl. 04.58 skrev Richard Larkin notifications@github.com:

@rasmuskreiner https://github.com/rasmuskreiner. I'm struggling to get the java class detected. How do you include you java class file in your buildozer.spec file? Using "android.add_src" or "android.add_jars"? Do you have to do anything extra to register this class on android? e.g. set the CLASSPATH? Thanks

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kivy/python-for-android/issues/1388#issuecomment-450537618, or mute the thread https://github.com/notifications/unsubscribe-auth/AFHUKzOjSnqhSHEifS0fPNyx4tBz2Xdqks5u-DoBgaJpZM4XEJmw.

[OptimusGREEN]

That java files don't always get copied into the build folder by buildozer. There's obviously a bug so need to manually copy them into the path that buildozer looks for them which will be in the build error, then it works ok. 

Sent from my Samsung Galaxy smartphone. -------- Original message --------From: rasmuskreiner notifications@github.com Date: 16/01/2019 19:36 (GMT+00:00) To: kivy/python-for-android python-for-android@noreply.github.com Cc: OptimusGREEN johnmcbridegreen@gmail.com, Mention mention@noreply.github.com Subject: Re: [kivy/python-for-android] Unable to open a zip on Android: zipfile.BadZipfile: File is not a zip file (#1388) Sorry the late response. I’m on paternity leave hence the slow reply rate…

As I recall I was also struggling with this part. Ultimately I ended up putting the java files in the root directory, and adding them to the “android.add_src". Eg. android.add_src = unzipFile.java.

Den 30. dec. 2018 kl. 04.58 skrev Richard Larkin notifications@github.com:

@rasmuskreiner https://github.com/rasmuskreiner. I'm struggling to get the java class detected. How do you include you java class file in your buildozer.spec file? Using "android.add_src" or "android.add_jars"? Do you have to do anything extra to register this class on android? e.g. set the CLASSPATH? Thanks

—

You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/kivy/python-for-android/issues/1388#issuecomment-450537618, or mute the thread https://github.com/notifications/unsubscribe-auth/AFHUKzOjSnqhSHEifS0fPNyx4tBz2Xdqks5u-DoBgaJpZM4XEJmw.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/kivy/python-for-android","title":"kivy/python-for-android","subtitle":"GitHub repository","main_image_url":"https://github.githubassets.com/images/email/message_cards/header.png","avatar_image_url":"https://github.githubassets.com/images/email/message_cards/avatar.png","action":{"name":"Open in GitHub","url":"https://github.com/kivy/python-for-android"}},"updates":{"snippets":[{"icon":"PERSON","message":"@rasmuskreiner in #1388: Sorry the late response. I’m on paternity leave hence the slow reply rate…\n\nAs I recall I was also struggling with this part. Ultimately I ended up putting the java files in the root directory, and adding them to the “android.add_src\". Eg. android.add_src = unzipFile.java.\n\n\u003e Den 30. dec. 2018 kl. 04.58 skrev Richard Larkin \u003cnotifications@github.com\u003e:\n\u003e \n\u003e @rasmuskreiner \u003chttps://github.com/rasmuskreiner\u003e. I'm struggling to get the java class detected. How do you include you java class file in your buildozer.spec file? Using \"android.add_src\" or \"android.add_jars\"? Do you have to do anything extra to register this class on android? e.g. set the CLASSPATH? Thanks\n\u003e \n\u003e —\n\u003e You are receiving this because you were mentioned.\n\u003e Reply to this email directly, view it on GitHub \u003chttps://github.com/kivy/python-for-android/issues/1388#issuecomment-450537618\u003e, or mute the thread \u003chttps://github.com/notifications/unsubscribe-auth/AFHUKzOjSnqhSHEifS0fPNyx4tBz2Xdqks5u-DoBgaJpZM4XEJmw\u003e.\n\u003e \n\n"}],"action":{"name":"View Issue","url":"https://github.com/kivy/python-for-android/issues/1388#issuecomment-454910631"}}}

[

{

"@context": "http://schema.org",

"@type": "EmailMessage",

"potentialAction": {

"@type": "ViewAction",

"target": "https://github.com/kivy/python-for-android/issues/1388#issuecomment-454910631",

"url": "https://github.com/kivy/python-for-android/issues/1388#issuecomment-454910631",

"name": "View Issue"

},

"description": "View this Issue on GitHub",

"publisher": {

"@type": "Organization",

"name": "GitHub",

"url": "https://github.com"

}

}

]

[Zen-CODE]

Thanks both. The ".add_src" worked for me in the end. The files did get copied, but an import in the java class that was not available on Android made the loading fail, so the error was misleading.

[thica]

Having the same issue. unzip doesn't work, is_zipfile doesn't work as well (only Android: (Windows, Linux, OSX) works with the same code on the same zipfiles)

[brentpicasso]

Seeing the same exact issue here as well. Is there an alternate zipfile library for python we can use as a test?

I can access and copy the zip file around the Android system and verify it is not corrupt.

[brentpicasso]

I'm assuming the problem with python's built-in zipfile is that is relies on OS level capability, which causes the problem. It is not doing it in pure python code.

Know of any any pure-python archive managers we can use to side-step the issue?

Also looking at archiving using tar.gz as an alternative. Found patool, but it seems to use /system/bin/tar - I'd prefer it to not rely on it, but seems to work as a backup.

Will still try to get .zip files to work in an alternate way.

[brentpicasso]

So, I can get uncompressed tar files to work with patool, but not if it's .gz compressed. Unsure if it's related to this issue or not.

patool: Extracting /data/data/com.autosportlabs.racecapture/files/app/defaults/default_mappings.tar.gz ... 02-13 18:50:37.343 22614 23272 I python : [ERROR ] [PresetManager] Could not load default presets: Command `['/system/bin/tar', '--extract', '-z', '--file', '/data/data/com.autosportlabs.racecapture/files/app/defaults/default_mappings.tar.gz', '--directory', '/storage/emulated/0/Android/data/com.autosportlabs.racecapture/files/presets']' returned non-zero exit status 1 /data/data/com.autosportlabs.racecapture/files/app/defaults/default_mappings.tar.gz

[Zen-CODE]

@brentpicasso. So, I ended up writing a small Java class to handle extraction on Android and then using pyjnius to call that. I built that class with the same interface so I could just swap out the "ZipFile" implementation on Android. It's currently in a private repo, but I'd be happy to upload those file for you if you would like.

[thica]

Hi Zen, I would be interested as this is one reasons which blocks me from using the new build environment

[brentpicasso]

@Zen-CODE Thanks. I see the examples shared above and it sounds like it would work.

I understand this is a sub-optimal work around, especially so since we are supporting many different platforms besides Android.

I was looking for a pure python implementation of an unzip library, so we can use the same code everywhere without switching between platforms. So far they all seem to be wrappers for the standard zipfile python library. Any suggestions on this?

I'm assuming a pure python library should be able to extract the zip file if it doesn't touch any OS level functions.

[thica]

Hi Brent, for me it looks, that it is a Android specific problem, so you could capsulate it to distingish it for Android and the rest. Not optimal, but a valid work around.

[brentpicasso]

Hi, Yes, thank you, I already understand that it is Android specific. As I wrote in my post, I'd prefer a pure python implementation, if possible.

[brentpicasso]

Here is a helper function that will help you choose automatically. Just update it to reference the java class and function you are using, such as the one @OptimusGREEN suggested.

@Zen-CODE does it make sense at this point to include this wrapper in Kivy's utility library, or do you think we'll get Kivy on Android fixed for zip files so we won't need this workaround?

from kivy import platform
__all__ = ('extract_zipfile')

def extract_zipfile(zipfile_path, destination_dir):
    if platform == 'android':
        from jnius import autoclass
        extract = autoclass("MY JAVA CLASS")()
        extract.extractFile(zipfile_path, destination_dir)
    else:
        import zipfile
        with zipfile.ZipFile(zipfile_path, 'r') as z:
            z.extractall(destination_dir)

[Zen-CODE]

@brentpicasso. I doubt the dev's would want to to include that in kivy utils, as it adds extra complexity and only addresses one function of the rich ZipFile utility. It would be vastly preferable to solve the issue directly, but as ZipFile is not part of kivy, it's going to be hard to get the original developers to pay attention as it was not developed with Android in mind....

[thica]

I still don't know if it is a toolchain or a python issue. Python unzip did work in the past

[brentpicasso]

Trying to mentally get past how weird this is, but my guess is that:

• Python zipfile depends on some OS level library that is causing the breakage
• Java's implementation is pure (no JNI) and side-steps the issue.

My educated guess is, if there was a pure python zipfile implementation available as an alternate library, then that would side-step the issue just like Java does.

[Zen-CODE]

It worked when we built against api 19, min13, but broke when we switched to api 27, min 21. So it's almost certainly related to the api level. I'd guess a permissions issue, but that's just a guess. @brentpicasso. I would agree that a pure python implementation would probably side step it, but who would bother creating one when zipfile is a python standard library?

[brentpicasso]

Quite certain is is not a permission issue, as I was able to read and copy the zip file otherwise in python.

e.g. as a test, I was able to copy the .zip file to the destination folder, but I was not able to extract in place.

Agree it's silly to write an unzipping library to solve this problem, but if one existed already, it could be used.

[thica]

Just a remark: "is_zipfile" doesn't work as well, maybe the whole module doesn't work. I can confirm brent's remark, that it is unlikely a permission issue, as I tried several locations (and I do download the zip before I try to unzip them, and that works)

[brvier]

Got same issue with package_ressource as it s use zipfile for importing egg