iOS apps rejected by App Store

[freakboy3742]

Describe the bug

Via @johnfhima at https://github.com/beeware/Python-Apple-support/issues/175#issuecomment-1423064843

Apps packaged using this support package are being rejected by the App Store due to app structure.

Steps to reproduce

1. Generate a HelloWorld app for iOS
2. Push that app to TestFlight
3. See error "Invalid Bundle Structure - The binary file '...' is not permitted. Your app can’t contain standalone executables or libraries, other than the CFBundleExecutable of supported bundles."

Expected behavior

App should not be rejected by App Store.

Environment

• Operating System: iOS, possibly macOS
• Python version: all

Additional context

The error message referred to by the App Store links to this document. It suggests the

[@mhsmith edit: This error is also discussed at Technical Note TN2435]

A similar problem likely exists for macOS apps pushed to the App Store.

It's possible the app_packages and python-stdlib folders need to be moved to a "fake" Python framework package in a /Frameworks location.

[samschott]

Oh, that is annoying! I'm not particularly familiar with either iOS packaging requirements or Python builds, but would a "real" framework build work here? Similar to what the macOS Python installer installs in Library/Frameworks/Python.framework on macOS?

[freakboy3742]

That's definitely going to be my first pass at a solution. The problem that I foresee is that I'm don't know how aggressive App Store validation is on having "a single library with the same name as the framework" as a requirement. We need to have dozens of .so files, named as .so files, in specific directory structures. I certainly hope we don't need to process every Python module into it's own .framework so that iOS is happy...

[rickymohk]

I am also facing this issue. I tried putting all python stuffs into a framework, and use this framework in an app, but it still produces the error when validating. How does kivy pull this off?

[freakboy3742]

I have checked Kivy for a while, but last time I played with it, they solve the problem by avoiding it completely, and static compiling the binary modules. This approach is guaranteed to work, as all the library code in the main executable, but it prevents distributing packages as binary wheels.

[rickymohk]

I understand Kivy is merging the binaries in to the main executable, but I get confused when inspecting the app package content produced by Kivy. There are still .so files inside (e.g. in AppName.app/lib/python3.9/site-packages/) . So I wonder 1. why it still contains .so files when they already merge the binaries into the main exe and 2. why app store validation doesn't care about these .so files.

[freakboy3742]

Like I said, I haven't looked at Kivy for a while, so I don't know for sure. However, (a) are you sure they're included in the final published app bundle, not just the intermediate compilation artefacts; and (b) that Kivy apps are accepted by the iOS App Store? (b) in particular doesn't seem especially likely, but it would be an explanation.

[rickymohk]

After further inspection, turns out that the .so files packaged in the Kivy app are empty files, probably just placeholders. They use a custom importer to redirect the .so import for precompiled modules. Now I am able to use a Kivy generated XCode project as a barebone (without compiling the kivy recipe, only compile python and numpy), rename its main function and expose by a bridging header, then add my Swift files with new entry point and start develop with Swift and PythonKit from there. This way it passes the app store validation and successfully uploaded to TestFlight for internal testing. Though I haven't tried to submit for review yet.

[johnfhima]

Hi @rickymohk, could you provide some documentation/additional explanation about how can I implement your solution?

This would be really helpful.

[rickymohk]

Hi @rickymohk, could you provide some documentation/additional explanation about how can I implement your solution?

This would be really helpful.

Forgive me if it is not suitable to talk too deep into Kivy in this BeeWare thread. Here is how I did it:

1. Follow https://kivy.org/doc/stable/guide/packaging-ios-prerequisites.html#packaging-ios-prerequisites
2. Run pip install kivy-ios
3. Create a project folder and cd into it
4. Run toolchain build numpy
5. Create a dummy app folder(e.g. dummyapp) with a file named "main.py"
6. Run toolchain create AppName ./dummyapp An XCode project folder named "appname-ios" will be created
7. Open the created XCode project (e.g. appname-ios/appname.xcodeproj)
8. Select signing team
9. Remove the "Classes" folder (with bridge.m and bridge.h)
10. Edit "main.m"
    1. Extract the necessary initialization code (set PYTHONHOME, PYTHONPATH, load_custom_builtin_importer(), Py_Initialize(), etc.) from int main(int argc, char *argv[]) into a void initPython()
       1. Don't call Py_Finalize() since it breaks numpy (It throws Python exception: No module named '_posixsubprocess' when using numpy if Py_Finalize() is called. I figured this out by trial, don't know exactly what it does or its impact. Will be happy if someone could explain it)
    2. Remove int main(int argc, char *argv[]) and void export_orientation();
    3. Remove unused headers, leave only "Foundation.h" and "Python.h"
11. Create "main.h", write void initPython(); into it
12. Add Swift files (e.g. AppName.swift) as new entry point (with @main in it, such as your SwfitUI App, or AppDelegate if you use Storyboard), with create bridging header file
13. Write #import "main.h" in the bridging header (e.g. appname-Bridging-Header.h)
14. You now have a Swift project with Python environment and numpy support. Call initPython() in your entry point then use PythonKit to invoke Python codes.
15. Do the first bullet in https://github.com/kivy/kivy-ios/issues/409#issuecomment-623471112 before uploading to TestFlight

[johnfhima]

@rickymohk Thanks a lot for your help. I am trying to follow your instruction but I don't really know what is the necessary part of the main function that should be moved to the initPython function.

In case you feel it is not appropriate to give more information about ivy on this thread maybe you could send it to me by email (in my GitHub bio).

Thanks a lot again, it is few weeks I try to find a solution

[rickymohk]

@rickymohk Thanks a lot for your help. I am trying to follow your instruction but I don't really know what is the necessary part of the main function that should be moved to the initPython function.

In case you feel it is not appropriate to give more information about ivy on this thread maybe you could send it to me by email (in my GitHub bio).

Thanks a lot again, it is few weeks I try to find a solution

This is how my void iniyPython() ending up

void initPython()
{
    NSAutoreleasePool * pool = [[NSAutoreleasePool alloc] init];

    // Change the executing path to YourApp
    chdir("YourApp");

    // Special environment to prefer .pyo, and don't write bytecode if .py are found
    // because the process will not have a write attribute on the device.
    putenv("PYTHONOPTIMIZE=2");
    putenv("PYTHONDONTWRITEBYTECODE=1");
    putenv("PYTHONNOUSERSITE=1");
    putenv("PYTHONPATH=.");
    putenv("PYTHONUNBUFFERED=1");
    putenv("LC_CTYPE=UTF-8");
    // putenv("PYTHONVERBOSE=1");
    // putenv("PYOBJUS_DEBUG=1");

    // IOS_IS_WINDOWED=True disables fullscreen and then statusbar is shown
    putenv("IOS_IS_WINDOWED=False");

    NSString * resourcePath = [[NSBundle mainBundle] resourcePath];
    NSString *python_home = [NSString stringWithFormat:@"PYTHONHOME=%@", resourcePath, nil];
    putenv((char *)[python_home UTF8String]);

    NSString *python_path = [NSString stringWithFormat:@"PYTHONPATH=%@:%@/lib/python3.9/:%@/lib/python3.9/site-packages:.", resourcePath, resourcePath, resourcePath, nil];
    putenv((char *)[python_path UTF8String]);

    NSString *tmp_path = [NSString stringWithFormat:@"TMP=%@/tmp", resourcePath, nil];
    putenv((char *)[tmp_path UTF8String]);

    NSLog(@"Initializing python");
    Py_Initialize();

    // If other modules are using the thread, we need to initialize them before.
//    PyEval_InitThreads();

    // Add an importer for builtin modules
    load_custom_builtin_importer();

//    Py_Finalize();
    NSLog(@"Leaving");

    [pool release];
}

[johnfhima]

@rickymohk Did you succeed to install cocoa pods dependencies? Trying to install TensorFlow lite but it leads to several errors (while in a non ivy project it is working fine).

[ydesgagn]

@freakboy3742 let's try to talk 15 minutes this week. This is easy to fix.

1. We need to fix this repo so the libs are built with -shared instead of -bundle for all Apple SDKs. We need to update the patch files for all branches and I'm not too sure how to do it. When I tested it, there is always something failing, so clearly I was not doing the proper thing. Currently when you run file math.cpython-310-iphoneos.so you have Mach-O 64-bit bundle arm64. After the change, it should be Mach-O 64-bit dynamically linked shared library arm64
2. Apple doesn't allow mixing lib types in the same framework (.a and .so/.dylib) so we need to create a second framework with the content of lib-dynload or many frameworks. The load time of an iOS app is proportional to the number of dynamically linked shared libraries loaded so might be best to let the users add only the frameworks that they really need.

I have a bash script I can give you to create the frameworks above.

Ping me and we can align to close this.

Briefcase will also have to be changed when downloading pre-built wheels to extract the .so and create the frameworks on the fly (same bash script can be used as above).

[freakboy3742]

@ydesgagn I'm in Utah for PyCon US at the moment; so catching up should be a lot easier than it would normally be. I'll drop you a message on Discord to coordinate

[lin-it]

There is another problem I found that .so cannot be used in the online package of appstore

[freakboy3742]

FYI - The PRs that have been linked to this ticket are sufficient to resolve the problem; Travel Tips has been updated and published using pre-release versions of these PRs: https://apps.apple.com/au/app/travel-tips/id1336372310?platform=iphone

[KodaKoder]

Will the dylib update / App Store support be merged on the main or we have to use the dylib branch? Thanks

[freakboy3742]

@KodaKoder It will be merged - we just need to get everything polished first. There's a lot of moving pieces that we need to be finalised:

• Support package build changes to produce wheels in the new dylib format
• Briefcase template changes to accomodate the new dylib format
• Changes to Briefcase to modify the process of deploying wheels
• Updates to the wheel building tools to generate new-style dylibs.
• Actually build and publish new-style dylib wheels.

We're also using point (4) as an opportunity to rework how we build dylib wheels, integrating crossenv rather than the ad hoc approach we've used to date.

This is my personal top priority at the moment; I'm hoping to have things finalised by the end of the month.

[ydesgagn]

@freakboy3742 let me know if I can help with something.

[KodaKoder]

I will test the branch today with a little an app I will report if I encounter any problems

[freakboy3742]

@KodaKoder You may want to wait until I've flagged the PRs as ready for review before starting on that testing. I'm still making changes that will impact on your ability to test this code.