PIL (pillow) loading fails with pkg_resources error

[Zen-CODE]

Versions

• Python : 3.8
• MacOS version : 10.15.5
• XCode Version : 11.2.1
• Cython version : 0.28.1

Describe the bug When running the program, the loading fails with a missing module error dispalyed in the console.

   File "/Users/richard/Library/Developer/CoreSimulator/Devices/E5F90713-0057-4F0B-A728-F6F27038B40F/data/Containers/Bundle/Application/8522E39B-C6F1-441E-B036-71CAF7A37FCE/btp-kivy-app.app/lib/python3.8/site-packages/Pillow-6.1.0-py3.8-macosx-10.15-x86_64.egg/PIL/_imaging.py", line 3, in __bootstrap__
 ModuleNotFoundError: No module named 'pkg_resources'
2020-07-15 20:05:34.507524+0200 btp-kivy-app[47771:3173351] Application quit abnormally!

To Reproduce toolchain build pillow

Logs

[INFO   ] [Audio       ] Providers: audio_sdl2 (audio_avplayer, audio_ffpyplayer ignored)
 Traceback (most recent call last):
   File "/Users/richard/Repos/btp/kivy-ios/btp-kivy-app-ios/YourApp/main.py", line 6, in <module>
   File "/Users/richard/Repos/btp/kivy-ios/btp-kivy-app-ios/YourApp/controller.py", line 6, in <module>
   File "/Users/richard/Repos/btp/kivy-ios/btp-kivy-app-ios/YourApp/lib/CPData/lifestory/game.py", line 25, in <module>
   File "/Users/richard/Repos/btp/kivy-ios/btp-kivy-app-ios/YourApp/lib/CPData/lifestory/lsmanager.py", line 4, in <module>
   File "/Users/richard/Repos/btp/kivy-ios/btp-kivy-app-ios/YourApp/lib/CPData/lifestory/lsicon.py", line 6, in <module>
   File "/Users/richard/Repos/btp/kivy-ios/btp-kivy-app-ios/YourApp/lib/CPData/game/exifrotation.py", line 15, in <module>
   File "<frozen zipimport>", line 259, in load_module
   File "/Users/richard/Library/Developer/CoreSimulator/Devices/E5F90713-0057-4F0B-A728-F6F27038B40F/data/Containers/Bundle/Application/8522E39B-C6F1-441E-B036-71CAF7A37FCE/btp-kivy-app.app/lib/python3.8/site-packages/Pillow-6.1.0-py3.8-macosx-10.15-x86_64.egg/PIL/Image.py", line 95, in <module>
   File "<frozen zipimport>", line 259, in load_module
   File "/Users/richard/Library/Developer/CoreSimulator/Devices/E5F90713-0057-4F0B-A728-F6F27038B40F/data/Containers/Bundle/Application/8522E39B-C6F1-441E-B036-71CAF7A37FCE/btp-kivy-app.app/lib/python3.8/site-packages/Pillow-6.1.0-py3.8-macosx-10.15-x86_64.egg/PIL/_imaging.py", line 7, in <module>
   File "/Users/richard/Library/Developer/CoreSimulator/Devices/E5F90713-0057-4F0B-A728-F6F27038B40F/data/Containers/Bundle/Application/8522E39B-C6F1-441E-B036-71CAF7A37FCE/btp-kivy-app.app/lib/python3.8/site-packages/Pillow-6.1.0-py3.8-macosx-10.15-x86_64.egg/PIL/_imaging.py", line 3, in __bootstrap__
 ModuleNotFoundError: No module named 'pkg_resources'
2020-07-15 20:05:34.507524+0200 btp-kivy-app[47771:3173351] Application quit abnormally!
2020-07-15 20:05:34.544800+0200 btp-kivy-app[47771:3173351] Leaving

[Zen-CODE]

Turns out there are plenty of references to pkg_resources in the codebase. Not sure where they are all coming from, but it's enough to warrant bringing pkg_resources back until these are gone. There to no references in the pillow library itself, interestingly enough....

[Zen-CODE]

When trying to upgrade to pillow 7.2.0 as a test, we get this warning.

Following up in this discussion: https://github.com/python-pillow/Pillow/issues/4730

[AndreMiras]

I think adding setuptools to python_depends should be enough as setuptools ships pkg_resources. Let's give that a try

[Zen-CODE]

setuptools is already in the python_depends for pillow?

[AndreMiras]

Oh yes indeed, then there's something a bit off. I haven't tried Pillow yet, but this is an issue you get on device straight after trying to do the import e.g. from PIL import Image?

[Zen-CODE]

Correct.

[Zen-CODE]

After re-instating pkgresources and building that, I get the follow error in the pillow build.

  STDOUT:
Traceback (most recent call last):
  File "setup.py", line 20, in <module>
    from setuptools import Extension, setup
  File "/Users/richard/Repos/btp/kivy-ios/dist/hostpython3/lib/python3.8/site-packages/setuptools/__init__.py", line 18, in <module>
    import setuptools.version
  File "/Users/richard/Repos/btp/kivy-ios/dist/hostpython3/lib/python3.8/site-packages/setuptools/version.py", line 1, in <module>
    import pkg_resources
  File "/Users/richard/Repos/btp/kivy-ios/dist/root/python3/lib/python3.8/site-packages/pkg_resources.py", line 1548, in <module>
    register_loader_type(importlib_bootstrap.SourceFileLoader, DefaultProvider)
AttributeError: module 'importlib._bootstrap' has no attribute 'SourceFileLoader'

[Zen-CODE]

As an experiment, tried making this change in pkgresources as another user suggested.

-    import importlib._bootstrap as importlib_bootstrap
+    import importlib._bootstrap_external as importlib_bootstrap

But then get this error. Does not seem like valid change.

  STDOUT:
Traceback (most recent call last):
  File "/Users/richard/Repos/btp/kivy-ios/dist/root/python3/lib/python3.8/site-packages/pkg_resources.py", line 2643, in _dep_map
    return self.__dep_map
  File "/Users/richard/Repos/btp/kivy-ios/dist/root/python3/lib/python3.8/site-packages/pkg_resources.py", line 2505, in __getattr__
    raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map

[Zen-CODE]

@AndreMiras Any thoughts on the SourceFileLoader error? It kinda baffles me that this used to work? I can't really see what's changed that would break it?....

[Zen-CODE]

Could it be related to this? It seems the pip command no longer works?

(venv) richard@Richards-MacBook-Pro kivy-ios % python toolchain.py pip install setuptools 
[INFO    ] Include dir added: {arch.arch}/freetype
[INFO    ] Include dir added: {arch.arch}/hostlibffi
[INFO    ] Global: hostpython located at /Users/richard/Repos/btp/kivy-ios/dist/hostpython3/bin/python
[INFO    ] Global: hostpgen located at /Users/richard/Repos/btp/kivy-ios/dist/hostpython3/bin/pgen
[INFO    ] Include dir added: {arch.arch}/ffi
[INFO    ] Include dir added: {arch.arch}/libjpeg
[INFO    ] Include dir added: {arch.arch}/openssl
[INFO    ] Include dir added: common/sdl2
[INFO    ] Include dir added: common/sdl2_image
[INFO    ] Include dir added: common/sdl2_mixer
[INFO    ] Include dir added: common/sdl2_ttf
[ERROR   ] Executing pip with: ['install', '--isolated', '--prefix', '/Users/richard/Repos/btp/kivy-ios/dist/root/python3', 'setuptools']
Traceback (most recent call last):
  File "toolchain.py", line 3, in <module>
    main()
  File "/Users/richard/Repos/btp/kivy-ios/kivy_ios/toolchain.py", line 1523, in main
    ToolchainCL()
  File "/Users/richard/Repos/btp/kivy-ios/kivy_ios/toolchain.py", line 1288, in __init__
    getattr(self, args.command)()
  File "/Users/richard/Repos/btp/kivy-ios/kivy_ios/toolchain.py", line 1482, in pip
    _pip(sys.argv[2:])
  File "/Users/richard/Repos/btp/kivy-ios/kivy_ios/toolchain.py", line 1182, in _pip
    pip_cmd = sh.Command(pip_path)
  File "/Users/richard/Repos/btp/kivy-ios/venv/lib/python3.7/site-packages/sh.py", line 1202, in __init__
    raise CommandNotFound(path)
sh.CommandNotFound: /Users/richard/Repos/btp/kivy-ios/dist/hostpython3/bin/pip3

[Zen-CODE]

@misl6 It seems the pip binary in missing from the dist/hostpython3/bin/? Do you perhaps know why? Thanks

[misl6]

@Zen-CODE Do you have evidences on when the pip3 disappeared?

I'm trying to download the latest CI logs to see if it produces some error, but currently GitHub is failing to produce such information 🤦 

[Zen-CODE]

I can't say when for sure, but I can show you that it is certainly missing compared to a previously working checkout & build of kivy-ios....

[misl6]

@Zen-CODE I've rebuilt kivy-ios==master twice on my setup and pip3 it's available. Can you also try to rebuild from scratch and capture the logs for further investigation?

[Zen-CODE]

Yes, the latest master does build pip, my apologies. But the pillow build still fails. I'm busy trawling through the commits to find the last one that gives a successful pillow build...;-(