search

beeware / Python-Apple-support

A meta-package for building a version of Python that can be embedded into a macOS, iOS, tvOS or watchOS project.
MIT License
1.12k stars 161 forks source link

Switch to using .dylib format for iOS binary modules #187

Closed freakboy3742 closed 1 year ago

freakboy3742 commented 1 year ago

Fixes #176.

The iOS App Store is rejecting apps that include binary components in the "Resources" folder. They will apparently only accept dynamically loaded components if they're (a) packaged as Frameworks, and (b) contained in the Frameworks folder.

This PR (which is is substantially built on the work of @alibeyram, @landrybr and team) allows for this usage pattern. It doesn't change how the support package is distributed; it relies on the project using the support package to move the .dylibs into their final location.

It incorporates changes that:

PR Checklist:

freakboy3742 commented 1 year ago

At this point, the code requires the use of the template provided by @alibeyram: https://github.com/beeware/briefcase-iOS-Xcode-template/pull/22

There's one more change required - dylibs need some name munging to avoid collisions when two submodules use the same binary name.

alibeyram commented 1 year ago

Thanks for the credit. Let me tag @landrybr who has done most of this.

@freakboy3742 I have a question about the AppleFrameworkFinder. Have you tested it with app_packages? I think sys.meta_path.append( will and I used sys.meta_path.insert(0, to go around the issue I am gonna mention.

For example 

from . import _imaging as core

Given there are other meta_path that are priority they will try to load the _imaging.cpython-310-iphonesimulator.dylib which will not be available in . (current directory)

Regarding the name collisions, I think we can assume all the binaries (.dylib) are at the top directory in the packages. If you are ok with this assumption I can quickly modify both template and the AppleFrameworkFinder

freakboy3742 commented 1 year ago

Thanks for the credit. Let me tag @landrybr who has done most of this.

@freakboy3742 I have a question about the AppleFrameworkFinder. Have you tested it with app_packages? I think sys.meta_path.append( will and I used sys.meta_path.insert(0, to go around the issue I am gonna mention.

I'm still in the process of testing; from what testing I've done so far, the only difference between insert(0, ...) and append(...) is whether the standard Python loader is tried first or last (and thus whether an attempt to load a pure-python package has to try 3 non-existent binary dylibs before succeeding).

However, I'll make sure testing deep packages is part of my testing before I finalise this patch.

alibeyram commented 1 year ago

Ok seems like I made some silly error. append seems to work fine.

mhsmith commented 1 year ago

installs a bootstrap loader to look in the Frameworks folder for .dylib modules

Regardless of where it's loaded from, the module's __file__ should be set to its "original" location within the Python directory tree, because this attribute is often used to find other files with relative paths. In my experience on Android, the fact that the __file__ doesn't actually exist has never caused a problem. But the attribute does need to be set before any code in the module is executed, i.e. between the "create" and "exec" stages in the importer.

There's one more change required - dylibs need some name munging to avoid collisions when two submodules use the same binary name.

The example I encountered on Android was "utils", which can be found at cytoolz.utils, h5py.utils and zmq.backend.cython.utils.

I didn't keep a note of the module name, but I also encountered a case in TensorFlow where one Python module was dynamically linked against another one. With ELF binaries, such links are resolved using the DT_NEEDED and DT_SONAME values compiled into the binaries, so the filename they're loaded from doesn't matter. But I don't know whether Mach-O has a similar mechanism.

For all of the above issues, it's also worth checking whether Apple allows you to load a module from the Frameworks folder via a symlink in a different location.

freakboy3742 commented 1 year ago

installs a bootstrap loader to look in the Frameworks folder for .dylib modules

Regardless of where it's loaded from, the module's __file__ should be set to its "original" location within the Python directory tree, because this attribute is often used to find other files with relative paths. In my experience on Android, the fact that the __file__ doesn't actually exist has never caused a problem. But the attribute does need to be set before any code in the module is executed, i.e. between the "create" and "exec" stages in the importer.

Noted - thanks for the heads up.

I didn't keep a note of the module name, but I also encountered a case in TensorFlow where one Python module was dynamically linked against another one. With ELF binaries, such links are resolved using the DT_NEEDED and DT_SONAME values compiled into the binaries, so the filename they're loaded from doesn't matter. But I don't know whether Mach-O has a similar mechanism.

It does; I'll need to dig into what RPATHs need to be rewritten to support this.

For all of the above issues, it's also worth checking whether Apple allows you to load a module from the Frameworks folder via a symlink in a different location.

That's a good idea - unfortunately, it's taken me most of the day to confirm that the answer is... no. The Xcode app validator doesn't appear to differentiate between a bare dylib, and a symlink to a bare dylib.

alibeyram commented 1 year ago

For all of the above issues, it's also worth checking whether Apple allows you to load a module from the Frameworks folder via a symlink in a different location.

We tried that first and it did not work.

alibeyram commented 1 year ago

@freakboy3742 let me know if you decided to go with 

framework_name = "_".join(fullname.split("."))

to solve the collision issue so I can update the template with the correct bash script.

freakboy3742 commented 1 year ago

@alibeyram I've just opened beeware/briefcase-iOS-Xcode-template#23. In addition to the framework naming change, I needed to make some tweaks that I wanted to make around how the script executes on repeated passes, plus some simplifications on how the processing step is invoked to optimise compilation time.

freakboy3742 commented 1 year ago

By way of proof - I've pushed up an update to Travel Tips to the App Store. It's past initial validation and been accepted for TestFlight; now the wait begins to see if human reviewers like it.

Next step is to confirm that third-party libraries can be compiled and used.

mhsmith commented 1 year ago

Next step is to confirm that third-party libraries can be compiled and used.

You mentioned today that this means all third-party libraries would have to be recompiled into new wheels containing dylibs, which couldn't be loaded by older versions of the support package which don't have the custom finder. Is this just because of their filenames? If so, what if the new wheels kept on using the .so suffix, and then Briefcase renamed them at the same time as it moved them into frameworks?

alibeyram commented 1 year ago

It's not just a rename. The validator will not accept the .so (shared object) architecture. You can run otool to see the differences between so and dylib

mhsmith commented 1 year ago

I understand that, but the question is, how do we release the new wheel files into the repository without breaking all the builds of existing apps using old Briefcase versions? Those users may be happy enough to have a working app even if they can't release it on the App Store, so it's not like those old versions of Briefcase were useless to them, and we shouldn't force them to upgrade without warning.

alibeyram commented 1 year ago

I see your point now.

freakboy3742 commented 1 year ago

Four possible solutions to the backwards incompatibility problem:

  1. The nuclear option: Delete all the old packages, and start from scratch.
  2. Create a new Anaconda repo for "new style" dylib packages, and update Briefcase to use that repo.
  3. Anaconda.org provides an option for "labels"; this is effectively a tagging mechanism, but the tags are each exposed as a separate PyPI endpoint. By default the "main" tag appears on https://pypi.anaconda.org/beeware/simple, but if we were to add a dylib label to the new packages, the repo https://pypi.anaconda.org/beeware/label/dylib/simple would only serve dylib-compatible packages. As long as we don't tag new packages with main, we should be safe.
  4. Use a different binary tag. All the wheels are currently tagged ios_12_0, and are multi-platform wheels. If we switch to using separate iphoneos_12_0 and iphonesimulator_12_0 wheels, and do the merge in Briefcase, it means creating wheels becomes a lot simpler (as there's no post-processing step required). For this to be the best option, we'd need to be able to produce multi-arch iphonesimulator wheels in a single pass - I think this might be possible, but I need to do more tinkering.
mhsmith commented 1 year ago

When we talked about this the other day, we agreed that the old version of the support package probably would be able to load dylibs if they were called .so, but the .so -> MH_BUNDLE / .dylib -> MH_DYLIB mapping is so well established on Apple platforms that deviating from it would cause confusion.

So we concluded the best option was "4. Use a different binary tag." And we were inclined towards using separate wheels for each of the 3 ABIs, because:

As for the format of the tag, I suggest we use the same format as is already used by macOS, Linux and Android, with the architecture after the version number. In which case, it makes sense to treat "simulator" as part of the architecture:

mhsmith commented 1 year ago

we were inclined towards using separate wheels for each of the 3 ABIs

The downside of this is that Briefcase would have to run pip multiple times, and either merge the results together, or store them side by side.

However, all of that could be avoided if Briefcase just knew whether it was building for a physical or simulated device, because iOS currently only supports one physical architecture, and there is in practice only one useful simulator architecture – the architecture of the build machine.

alibeyram commented 1 year ago

Any downside to doing option 3 and changing the anaconda tag going forward for future versions of the briefcase? I see the benefits of option 4 specifically if other projects (e.g. Kivy) want to use this approach as well

freakboy3742 commented 1 year ago

Any downside to doing option 3 and changing the anaconda tag going forward for future versions of the briefcase? I see the benefits of option 4 specifically if other projects (e.g. Kivy) want to use this approach as well

There's no specific downside to introducing a new tag - in fact, it might be a good idea to do in addition to option 4 to avoid confusion over which packages are available. The bigger question is whether there's a downside to having a single wheel, vs multiple per-platform (or per-architecture) wheels. I'm continuing to poke around in this space to get a working end-to-end toolchain that can build binary wheels with crossenv, then deploy them with Briefcase.

KodaKoder commented 1 year ago

The merge is very nice! just one thing trying to compile for Mac Catalyst is failing Do you plan to add the support?

freakboy3742 commented 1 year ago

@KodaKoder See #117.

It's not on our immediate roadmap, but if someone wants to add support, I'll take a look at a patch.

KodaKoder commented 1 year ago

Signing the libs raise this error

https://d.pr/i/LdU5EQ

freakboy3742 commented 1 year ago

@KodaKoder Thanks for the report. What did you do to reproduce this error? In particular, are you using the new Xcode template and Briefcase version?

KodaKoder commented 1 year ago

I just imported the framework and stlib following the guide https://github.com/beeware/Python-Apple-support/blob/main/USAGE.md and changed the script to sign dylib instead of so

freakboy3742 commented 1 year ago

Ah - that guide will need to be updated.

As the guide says at the top, the officially supported mechanism for using this package is with Briefcase. The new template (with new signing scripts) is beeware/briefcase-iOS-Xcode-template#23