Can't use AsyncImage with HTTPS URL (or any HTTPS url wit any request): fix is to manually load certifi

Sirfanas commented 5 years ago

Versions

Python: 3
OS: Android
Kivy: 1.10.1
Cython: 0.29.7

Description

Try to open HTTPS Url Failed with urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate Actually happening on Async Image I use like that:

AsyncImage:
    source: 'https://i.goopics.net/27Odx.png'

Work perfectly on Windows, not on Android

buildozer.spec

Command:

buildozer android debug

Spec file:

[app]

# (str) Title of your application
title = myapp

# (str) Package name
package.name = myapp

# (str) Package domain (needed for android/ios packaging)
package.domain = org.myapp

# (str) Source code where the main.py live
source.dir = ./kivy_app

# (list) Source files to include (let empty to include all the files)
source.include_exts = py,png,jpg,kv,atlas

# (list) List of inclusions using pattern matching
#source.include_patterns = assets/*,images/*.png

# (list) Source files to exclude (let empty to not exclude anything)
#source.exclude_exts = spec

# (list) List of directory to exclude (let empty to not exclude anything)
#source.exclude_dirs = tests, bin

# (list) List of exclusions using pattern matching
#source.exclude_patterns = license,images/*/*.jpg

# (str) Application versioning (method 1)
version = 0.2

# (str) Application versioning (method 2)
# version.regex = __version__ = ['"](.*)['"]
# version.filename = %(source.dir)s/main.py

# (list) Application requirements
# comma separated e.g. requirements = sqlite3,kivy
requirements = certifi,openssl,python3,kivy,android

# (str) Custom source folders for requirements
# Sets custom source for any requirements with recipes
# requirements.source.kivy = ../../kivy

# (list) Garden requirements
#garden_requirements =

# (str) Presplash of the application
#presplash.filename = %(source.dir)s/data/presplash.png

# (str) Icon of the application
#icon.filename = %(source.dir)s/data/icon.png

# (str) Supported orientation (one of landscape, sensorLandscape, portrait or all)
orientation = all

# (list) List of service to declare
#services = NAME:ENTRYPOINT_TO_PY,NAME2:ENTRYPOINT2_TO_PY

#
# OSX Specific
#

#
# author = © Copyright Info

# change the major version of python used by the app
osx.python_version = 3.7

# Kivy version to use
osx.kivy_version = 1.10.1

#
# Android specific
#

# (bool) Indicate if the application should be fullscreen or not
fullscreen = 0

# (string) Presplash background color (for new android toolchain)
# Supported formats are: #RRGGBB #AARRGGBB or one of the following names:
# red, blue, green, black, white, gray, cyan, magenta, yellow, lightgray,
# darkgray, grey, lightgrey, darkgrey, aqua, fuchsia, lime, maroon, navy,
# olive, purple, silver, teal.
#android.presplash_color = #FFFFFF

# (list) Permissions
android.permissions = INTERNET

# (int) Target Android API, should be as high as possible.
android.api = 27

# (int) Minimum API your APK will support.
android.minapi = 21

# (str) Android NDK version to use
android.ndk = 17c

# (int) Android NDK API to use. This is the minimum API your app will support, it should usually match android.minapi.
android.ndk_api = 21

# (bool) Use --private data storage (True) or --dir public storage (False)
#android.private_storage = True

# (str) Android NDK directory (if empty, it will be automatically downloaded.)
#android.ndk_path =

# (str) Android SDK directory (if empty, it will be automatically downloaded.)
#android.sdk_path =

# (str) ANT directory (if empty, it will be automatically downloaded.)
#android.ant_path =

# (bool) If True, then skip trying to update the Android sdk
# This can be useful to avoid excess Internet downloads or save time
# when an update is due and you just want to test/build your package
android.skip_update = False

# (bool) If True, then automatically accept SDK license
# agreements. This is intended for automation only. If set to False,
# the default, you will be shown the license when first running
# buildozer.
android.accept_sdk_license = True

# (str) Android entry point, default is ok for Kivy-based app
#android.entrypoint = org.renpy.android.PythonActivity

# (list) Pattern to whitelist for the whole project
#android.whitelist =

# (str) Path to a custom whitelist file
#android.whitelist_src =

# (str) Path to a custom blacklist file
#android.blacklist_src =

# (list) List of Java .jar files to add to the libs so that pyjnius can access
# their classes. Don't add jars that you do not need, since extra jars can slow
# down the build process. Allows wildcards matching, for example:
# OUYA-ODK/libs/*.jar
#android.add_jars = foo.jar,bar.jar,path/to/more/*.jar

# (list) List of Java files to add to the android project (can be java or a
# directory containing the files)
#android.add_src =

# (list) Android AAR archives to add (currently works only with sdl2_gradle
# bootstrap)
#android.add_aars =

# (list) Gradle dependencies to add (currently works only with sdl2_gradle
# bootstrap)
#android.gradle_dependencies =

# (list) Java classes to add as activities to the manifest.
#android.add_activites = com.example.ExampleActivity

# (str) python-for-android branch to use, defaults to master
#p4a.branch = master

# (str) OUYA Console category. Should be one of GAME or APP
# If you leave this blank, OUYA support will not be enabled
#android.ouya.category = GAME

# (str) Filename of OUYA Console icon. It must be a 732x412 png image.
#android.ouya.icon.filename = %(source.dir)s/data/ouya_icon.png

# (str) XML file to include as an intent filters in <activity> tag
#android.manifest.intent_filters =

# (str) launchMode to set for the main activity
#android.manifest.launch_mode = standard

# (list) Android additional libraries to copy into libs/armeabi
#android.add_libs_armeabi = libs/android/*.so
#android.add_libs_armeabi_v7a = libs/android-v7/*.so
#android.add_libs_x86 = libs/android-x86/*.so
#android.add_libs_mips = libs/android-mips/*.so

# (bool) Indicate whether the screen should stay on
# Don't forget to add the WAKE_LOCK permission if you set this to True
#android.wakelock = False

# (list) Android application meta-data to set (key=value format)
#android.meta_data =

# (list) Android library project to add (will be added in the
# project.properties automatically.)
#android.library_references =

# (list) Android shared libraries which will be added to AndroidManifest.xml using <uses-library> tag
#android.uses_library =

# (str) Android logcat filters to use
#android.logcat_filters = *:S python:D

# (bool) Copy library instead of making a libpymodules.so
#android.copy_libs = 1

# (str) The Android arch to build for, choices: armeabi-v7a, arm64-v8a, x86, x86_64
android.arch = armeabi-v7a

#
# Python for android (p4a) specific
#

# (str) python-for-android git clone directory (if empty, it will be automatically cloned from github)
#p4a.source_dir =

# (str) The directory in which python-for-android should look for your own build recipes (if any)
#p4a.local_recipes =

# (str) Filename to the hook for p4a
#p4a.hook =

# (str) Bootstrap to use for android builds
# p4a.bootstrap = sdl2

# (int) port number to specify an explicit --port= p4a argument (eg for bootstrap flask)
#p4a.port =

#
# iOS specific
#

# (str) Path to a custom kivy-ios folder
#ios.kivy_ios_dir = ../kivy-ios
# Alternately, specify the URL and branch of a git checkout:
ios.kivy_ios_url = https://github.com/kivy/kivy-ios
ios.kivy_ios_branch = master

# Another platform dependency: ios-deploy
# Uncomment to use a custom checkout
#ios.ios_deploy_dir = ../ios_deploy
# Or specify URL and branch
ios.ios_deploy_url = https://github.com/phonegap/ios-deploy
ios.ios_deploy_branch = 1.7.0

# (str) Name of the certificate to use for signing the debug version
# Get a list of available identities: buildozer ios list_identities
#ios.codesign.debug = "iPhone Developer: <lastname> <firstname> (<hexstring>)"

# (str) Name of the certificate to use for signing the release version
#ios.codesign.release = %(ios.codesign.debug)s

[buildozer]

# (int) Log level (0 = error only, 1 = info, 2 = debug (with command output))
log_level = 2

# (int) Display warning if buildozer is run as root (0 = False, 1 = True)
warn_on_root = 1

# (str) Path to build artifact storage, absolute or relative to spec file
# build_dir = ./.buildozer

# (str) Path to build output (i.e. .apk, .ipa) storage
# bin_dir = ./bin

#    -----------------------------------------------------------------------------
#    List as sections
#
#    You can define all the "list" as [section:key].
#    Each line will be considered as a option to the list.
#    Let's take [app] / source.exclude_patterns.
#    Instead of doing:
#
#[app]
#source.exclude_patterns = license,data/audio/*.wav,data/images/original/*
#
#    This can be translated into:
#
#[app:source.exclude_patterns]
#license
#data/audio/*.wav
#data/images/original/*
#

#    -----------------------------------------------------------------------------
#    Profiles
#
#    You can extend section / key with a profile
#    For example, you want to deploy a demo version of your application without
#    HD content. You could first change the title to add "(demo)" in the name
#    and extend the excluded directories to remove the HD content.
#
#[app@demo]
#title = My Application (demo)
#
#[app:source.exclude_patterns@demo]
#images/hd/*
#
#    Then, invoke the command line with the "demo" profile:
#
#buildozer --profile demo android debug

Logs

05-27 19:29:05.842 23309 23355 I python  : [ERROR  ] [Loader      ] Failed to load image <https://i.goopics.net/27Odx.png>
05-27 19:29:05.842 23309 23355 I python  : Traceback (most recent call last):
05-27 19:29:05.842 23309 23355 I python  :   File "/home/user/hostcwd/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/urllib/request.py", line 1317, in do_open
05-27 19:29:05.842 23309 23355 I python  :   File "/home/user/hostcwd/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/http/client.py", line 1229, in request05-27 19:29:05.842 23309 23355 I python  :   File "/home/user/hostcwd/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/http/client.py", line 1275, in _send_request
05-27 19:29:05.842 23309 23355 I python  :   File "/home/user/hostcwd/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/http/client.py", line 1224, in endheaders
05-27 19:29:05.842 23309 23355 I python  :   File "/home/user/hostcwd/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/http/client.py", line 1016, in _send_output
05-27 19:29:05.842 23309 23355 I python  :   File "/home/user/hostcwd/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/http/client.py", line 956, in send
05-27 19:29:05.842 23309 23355 I python  :   File "/home/user/hostcwd/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/http/client.py", line 1392, in connect05-27 19:29:05.842 23309 23355 I python  :   File "/home/user/hostcwd/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/ssl.py", line 412, in wrap_socket
05-27 19:29:05.842 23309 23355 I python  :   File "/home/user/hostcwd/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/ssl.py", line 853, in _create
05-27 19:29:05.842 23309 23355 I python  :   File "/home/user/hostcwd/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/ssl.py", line 1117, in do_handshake
05-27 19:29:05.842 23309 23355 I python  : ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1051)
05-27 19:29:05.842 23309 23355 I python  :
05-27 19:29:05.842 23309 23355 I python  : During handling of the above exception, another exception occurred:
05-27 19:29:05.842 23309 23355 I python  :
05-27 19:29:05.842 23309 23355 I python  : Traceback (most recent call last):
05-27 19:29:05.842 23309 23355 I python  :   File "/home/user/hostcwd/.buildozer/android/platform/build/build/python-installs/kydoo/kivy/loader.py", line 342, in _load_urllib
05-27 19:29:05.842 23309 23355 I python  :   File "/home/user/hostcwd/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/urllib/request.py", line 525, in open
05-27 19:29:05.842 23309 23355 I python  :   File "/home/user/hostcwd/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/urllib/request.py", line 543, in _open05-27 19:29:05.842 23309 23355 I python  :   File "/home/user/hostcwd/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/urllib/request.py", line 503, in _call_chain
05-27 19:29:05.842 23309 23355 I python  :   File "/home/user/hostcwd/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/urllib/request.py", line 1360, in https_open
05-27 19:29:05.842 23309 23355 I python  :   File "/home/user/hostcwd/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl-sqlite3/armeabi-v7a__ndk_target_21/python3/Lib/urllib/request.py", line 1319, in do_open
05-27 19:29:05.842 23309 23355 I python  : urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1051)>

I actually found a """solution""" using:

import ssl
try:
    _create_unverified_https_context = ssl._create_unverified_context
except AttributeError:
    # Legacy Python that doesn't verify HTTPS certificates by default
    pass
else:
    # Handle target environment that doesn't support HTTPS verification
    ssl._create_default_https_context = _create_unverified_https_context

But using that in my main.py don't fix AsyncImage or any call in other py file

Any ideas ? Thank's

AndreMiras commented 5 years ago

Have you tried adding certifi to your requirements?

Sirfanas commented 5 years ago

Yes it's in the requirements list in buildozer's config

AndreMiras commented 5 years ago

Oh yes I overlooked that then I don't know. I know it's not the first time I saw such report, I would be looking into the existing report see if it help https://github.com/search?q=org%3Akivy+CERTIFICATE_VERIFY_FAILED&type=Issues

Sirfanas commented 5 years ago

Yeah I already look all report on kivy, buildozer and python for Android, of course on Google but problems comes from compilation (not my case as it build without problem) or people give the bypass ssl verification, but I "can't" do it cause I don't want to do it everytime I use something calling an URL with HTTPs, and I'll have to use it frequently So if there's no way I will do it but if there's a better solution I would take it I'm okay to bypass ssl verification but I'd like it global and not do it everytime Thank's

AndreMiras commented 5 years ago

Well it's a pretty unsatisfying (and risky) solution to me, but yes you could try to put it to your top level __init__.py so you don't have to call it every time.

Also could you please share the output of he following command:

find .buildozer/ -name "certifi*"

Sirfanas commented 5 years ago

Hi, So I try to put this ugly monster in my top level init but it didn't fix the problem...

So I execute the command and here is the output:

user@f2b3ca0a45cf:~/hostcwd$ find .buildozer/ -name "certifi*"
.buildozer/android/platform/build/dists/kydoo/_python_bundle/_python_bundle/site-packages/certifi
.buildozer/android/platform/build/dists/kydoo/_python_bundle/_python_bundle/site-packages/certifi-2019.3.9.dist-info
.buildozer/android/platform/build/build/python-installs/kydoo/certifi
.buildozer/android/platform/build/build/python-installs/kydoo/certifi-2019.3.9.dist-info
.buildozer/android/platform/build/build/venv/lib/python3.6/site-packages/pip/_vendor/certifi
.buildozer/android/platform/build/build/other_builds/openssl/armeabi-v7a__ndk_target_21/openssl1.1/doc/HOWTO/certificates.txt

Hope it will help you !

AndreMiras commented 5 years ago

Oh so you're running in the "official" Docker image are you? That may help reproducing the issue eventually.

Also I see something weird, on the buildozer.spec you shared we have package.name = myapp which seems to be different from what I see here in the paths kydoo. Maybe it's worth cleaning/rebuilding via a rm -rf .buildozer/.

Then other things to consider to help with debugging. Make the simplest reproduction case possible, e.g. see if it's also failing using the requests library:

import requests
response = requests.get('https://i.goopics.net/27Odx.png')
print(response.status_code)

So the idea here is to see if the problem comes from the way it's implemented in Kivy.

Then I would also try with a most known domain such as google.com. Here it's too see if it fails with authority, google.com is signed by Google Internet Authority which is signed by GlobalSign Root CA. And goopics.net is Let's Encrypt Authority which is signed by Builtin Object Token.

If both test fail, then we would have to dig a bit deeper and see how is requests interacting with certifi. The idea would be to see at which point the code execution take different branches on your device where it's not working vs on your laptop. One thing I can think of is the certifi/cacert.pem couldn't not be found on device somehow.

Sirfanas commented 5 years ago

Hi, I make a really simple test: I get the content of https://www.google.com/ Then I try to open an Image (https://media.makeameme.org/created/its-working-oyy433.jpg) with Async image Works perfectly on computer, still have SSL issues on Android Here is the code: https://github.com/Sirfanas/kivy-ssl

Maybe it can be easier for you to find the problem with that ?

Thank's !

mgc8 commented 5 years ago

I am having the exact same problem. I'm using Linux (Ubuntu 18.04) for development and compilation, and the app works perfectly fine in the normal environment, however when run from a mobile phone, the CA bundle cannot be found.

Adding certifi to the requirements list was the first thing I tried, however it had no effect.

I was able to solve the issue for normal UrlRequest by adding the "ca_file" parameter and actually forcing the "certifi.where()" bundle to be added there, and it worked nicely: https://stackoverflow.com/questions/55816099/kivy-urlrequest-with-https

Unfortunately, "AsyncImage" doesn't have a similar "ca_file" option (or is there such a mechanism available?) and requests fail with the same error: python : urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1051)>

Being unable to use HTTPS properly in Python3 has been the main reason for me not using Kivy seriously until now, and with the latest changes which finally made it a first-class citizen (deprecating Crystax), I was looking forward to picking this up again... Is there any way we can fix this, other than disabling CA validation which is not really an option?

Sirfanas commented 5 years ago

Hi,

I finally fix it ! The think is that putting certifi requirements did install it, but don't make usage of it. So we have to manually precise to Python we want it to use the certifi CA file. The only think to do is put that code in main.py :

import certifi
import os

# Here's all the magic !
os.environ['SSL_CERT_FILE'] = certifi.where()

Why SSL_CERT_FILE ? Because ssl use this env to get the CA file, by default it's None on Android, so manually put it let us use the certifi CA file.

And most of all, this is global to all our app ! We don't need to override any method or to call previous code again !

You can give a look here : https://github.com/Sirfanas/kivy-ssl It's a little sample of how to solve SSL Certification problem.

Hope it help you !

Sirfanas commented 5 years ago

Not closing it now, maybe it can be great to put it in documenation ?

AndreMiras commented 5 years ago

Nice investigation and findings :clap: I agree let's not close until we can make a clean fix. If we don't find a way to fix yes documenting it is also a way. I haven't spent time yet trying to reproduce. However I'm already using the requests library for some of my apps without any issue. Let's hope I can reproduce with the example you shared so I can compare why it's working in certain cases. Again well done for digging into it and sharing your findings :+1:

AndreMiras commented 5 years ago

Good news, I could reproduce with your code example :tada: @Sirfanas do you mind if I pull request to your repository to make the reproduction case a bit cleaner? Basically I want to change it to a unit test proving the point and cleaning a couple of things. Then we could tag a release and use it for reference until we fix it and add a unit test in our code base

Sirfanas commented 5 years ago

Hi, Yeah do everything you need ! If it can help it's good !

AndreMiras commented 5 years ago

So I dug a bit further and now it all makes sense. So I've never seen this issue before because I've always used the requests module. Not only the requests module can use certifi, but it also ships (until version v2.15.1) its own cert an can fallback to it. See requests/certs.py.

Another thing is by default openssl will look for certs in the OPENSSLDIR which is set at configure/compile time. See include/internal/cryptlib.h. However since we don't define it in our recipe, it seems to default to /usr/local/ssl/. This can verified with:

import ssl; ssl.get_default_verify_paths()

Which returns the following on Android:

DefaultVerifyPaths(cafile=None, capath=None, openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/usr/local/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/usr/local/ssl/certs')

So I though we could fix it at recipe level by simply pointing to Android shipped certs. But apparently we can't because the certs shipped are not in a format expected by openssl. See https://stackoverflow.com/questions/15375577/how-to-point-openssl-to-the-root-certificates-on-an-android-device

Then maybe we could try make the recipe hooks to certifi by default somehow. For instance by linking to it at configure time or by patching the module to set this environment variable like you did.

Sirfanas commented 5 years ago

Well, make some search, seems not to be too big different format, this link seem to convert Android CA file format (jks ?) to openssl format (pem) : https://www.example-code.com/python/jks_to_pem.asp

I don't know what's the best between using certifi's CA files or convert existing Android CA files.

Both works but second option may be harder than just set an environment variable. The main think is did it change anything to use certifi CA files instead of Android's one ?

Is it possible to preset the environment variable so we don't have to do it in code like I did ? And how to do it when using Buildozer ?

If it's not possible I think it can be really interesting to have the possibility to put environment variable in buidlozer's config !

AndreMiras commented 5 years ago

Yes I saw some examples to convert it, but it could be a bit cumbersome. The easiest option I can think of is eventually patching from p4a and injecting the SSL_CERT_FILE environment variable. Like you did, but at p4a level. So for instance we could patch the Python ssl module to:

check if SSL_CERT_FILE is defined in the env and if not
check if certifi is available and if so inject with:
os.environ['SSL_CERT_FILE'] = certifi.where()

Something like that I guess

mgc8 commented 5 years ago

Hello @Sirfanas, thank you very much for the investigation and the provided workaround!

I can confirm that adding "_os.environ['SSL_CERTFILE'] = certifi.where()" to the initialisation section of the app solved the issue nicely, HTTPS works now on devices and in the simulators I've tried (on Android).

For now this is a good solution, if there would be a way to have this done by default then of course all the better.

caver456 commented 4 years ago

I had a similar problem and the solution on this thread worked for me too. A UrlRequest to https://blahblah worked fine on Windows, but would return "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain" on Android. I had tried ca_file=certifi.where() as an argument to UrlRequest, that did not solve it (strange...); but, setting the env var as above did solve it! So, thanks all for your work on this!