Allow configuration of custom build params for macos Xcode builds in pyproject.toml

michelcrypt4d4mus commented 3 weeks ago

What is the problem or limitation you are having?

I would like a project configuration option in pyproject.toml that would allow me to configure arbitrary Xcode build params that briefcase will then inject into the .xcodeproj template generated by briefcase create macos Xcode (or maybe briefcase build macos Xcode, tbh i'm not sure).

for example i modified my templated project.pbxproj to add a param LSUIElement = YES (which makes the app not appear in the dock - suitable for menu bar apps, console, apps, etc) to the buildSettings, which in turn ends up configuring Info.plist for the app Xcode ends up building.

In project.pbxproj it comes out as the line INFOPLIST_KEY_LSUIElement = YES; in the buildSettings dictionary:

            buildSettings = {
                ARCHS = "$(ARCHS_STANDARD)";
                                <...snip...>
                INFOPLIST_KEY_LSUIElement = YES;
                                <...snip...>
                STRIP_INSTALLED_PRODUCT = NO;
                        }

Describe the solution you'd like

There's a few possibilities.

For common xcode build switches like LSUIElement it might make sense to have a full blown "human legible" pyproject.toml configuration option. Something like app_is_ui_element_only = true or show_in_dock = false or whatever; I have no great ideas or strong feelings on the subject.

There are of course tons of Xcode configuration options though and manually implementing them all as pyproject.toml options seems kind of like a Dark Path that could lead to a lot of unnecessary maintenance (x100 because Apple has a bad habit of silently deprecating or outright disabling things like Xcode environment variables and build params), so I was thinking that just an array named something like xcodeproject_build_settings in pyproject.toml with the raw key/value pair Xcode wants would be sufficient to cover everyone's needs in all cases without being all that difficult to use for someone sophisticated enough to be bundling a briefcase app.

So it could be something like this in pyproject.toml instead if I wanted to inject LSUIElement and LSBackgroundOnly as xcodebuild configurations:
```
xcodeproject_additional_build_settings = [
   "INFOPLIST_KEY_LSBackgroundOnly": "YES",  
   "INFOPLIST_KEY_LSUIElement": "YES"
]
```

You could allow True instead of YES though I think being "closer to the metal" of Xcode's project file format will make briefcase developers' lives significantly easier in terms of template maintenance down the line.

Describe alternatives you've considered

As described in https://github.com/beeware/briefcase/issues/1859 currently am just manually making these modifications to the xcodeproj files and keeping them in source control even though these files exist in the build/ directory hierarchy which presumably should need to be kept in version control and should be able to be recreated by briefcase create or briefcase build.

Additional context

There's some backstory on how I am using briefcase to build a bundle I later embed into another app on https://github.com/beeware/briefcase/issues/1859.

freakboy3742 commented 3 weeks ago

There's precedent for this sort of thing in the Android configuration - there's a bunch of configuration points that allow for injection of raw configuration items into the AndroidManifest.xml. There are also extension points for Info.plist and Entitlements.plist for macOS projects. I don't have a fundamental objection to adding analogous extension points into the Xcode project file for macOS (and/or iOS, for that matter). The issue with Xcode project files specifically is that they have a bunch of content already, so overwriting existing settings is going to be problematic.

Adding more "feature specific" extensions (like "show in dock") depends on being able to expose those in a meaningful cross-platform way. We're unlikely to add a setting to turn on a specific Xcode setting unless there's a reasonable interpretation of that setting for other platforms.

michelcrypt4d4mus commented 3 weeks ago

so overwriting existing settings is going to be problematic.

maybe just check for any collisions between the user's configured xcodeproject_additional_build_settings params and the plist data in the extant pbxproj and warn on any overwrites that might happen if that's a concern? i don't know much of anything about cookiecutter but with plistlib you can basically just write python code to handle .pbxproj files the same way you would handle any other python dictionary (and you can almost do the same thing in bash with /usr/libexec/PlistBuddy)

Adding more "feature specific" extensions (like "show in dock") depends on being able to expose those in a meaningful cross-platform way. We're unlikely to add a setting to turn on a specific Xcode setting

i agree it's not a good idea to do that. IMHO the raw xcodeproject_additional_build_settings approach is much more future proof compared trying to maintain a ton of random pyproject.toml config options that map to the underlying xcode env vars in custom ways.

i mean hell just look at Xcode itself - when you add params to Info.plist inside xcode's GUI you get a nice human readable string instead of the actual underlying string which is always something like LSUIElement. seems custom designed to help xcode users make dumb mistakes.

freakboy3742 commented 3 weeks ago

maybe just check for any collisions between the user's configured xcodeproject_additional_build_settings params and the plist data in the extant pbxproj and warn on any overwrites that might happen if that's a concern? i don't know much of anything about cookiecutter but with plistlib you can basically just write python code to handle .pbxproj files the same way you would handle any other python dictionary (and you can almost do the same thing in bash with /usr/libexec/PlistBuddy)

Except that the values you're describing aren't just "values for your Info.plist". The buildSettings section you're proposing for customisation also includes values like ARCHS and STRIP_INSTALLED_PRODUCT, so it would be possible for a user to define those values as well as part of your xcodeproject_additional_build_settings.

To support this, I suspect we would need to define a Jinja template field that can parse an Xcode settings block, do a dictionary merge with user-provided values, then output the merged settings.

I also want to point out (again) that if your goal is to add Info.plist keys specifically, there is an existing configuration point to do this. If you add:

info.LSBackgroundOnly = True
info.LSUIElement = True

to your app's configuration in the macOS section, these values will be injected into the generated app's Info.plist. This works for both the app and Xcode templates.

That doesn't remove the need for this feature, but it does remove one use case for it.

michelcrypt4d4mus commented 3 weeks ago

_so it would be possible for a user to define those values as well as part of your xcodeproject_additional_build_settings._

IMHO a reasonable solution would be to quote "with great power comes great responsibility" in the docs for such a feature and call it a day.

I also want to point out (again) that if your goal is to add Info.plist keys specifically,

yeah i caught that the first time so thanks for pointing it out - i think it does at least partially solve my Xcode build configuration situation.

michelcrypt4d4mus commented 3 weeks ago

just went back over it and the only customization i had to add to the pbxproj file that wasn't configurable with info in the pyproject.toml was this SKIP_INSTALL flag that apple's documentation claims is necessary for any application code embedded in another bundle. I have no idea what it does or if it actually matters but that apple documentation also provides a good example of how one has to do a very complicated dance around xcode's codesigning approach to get it to build and sign an app bundle with > 1 executable.

/* Begin XCBuildConfiguration section */
        BLAHBLAHXXXXXXD009178D1 /* Debug */ = {
            isa = XCBuildConfiguration;
            buildSettings = {
                                <...snip...>
                SKIP_INSTALL = YES;
            };
            name = Debug;
        };

freakboy3742 commented 3 weeks ago

Are you certain you actually need that setting? The documentation you reference mentions this being used to avoid installing the app into the archive - but Briefcase doesn't use the archive functionality of Xcode. AIUI, you only need to use that functionality if you're planning to distribute through the macOS App Store, which Briefcase doesn't currently support.

michelcrypt4d4mus commented 3 weeks ago

definitely not certain and actually highly skeptical that it's necessary. only reason it's there is because i was following that documentation when i setup my app initially. if it turns out to be unnecessary i'll report back.

which Briefcase doesn't currently support.

curious what you mean by this? is there something about the app bundle created by briefcase that makes it inherently impossible to submit to the app store or is it just that there would need to be some tweaking to get a briefcase app approved? curious bc while i'm initially planning to distribute my app outside the app store i still had at least some vague idea that eventually with some cleanup of entitlements etc. i could offer submit it.

freakboy3742 commented 3 weeks ago

which Briefcase doesn't currently support.

curious what you mean by this? is there something about the app bundle created by briefcase that makes it inherently impossible to submit to the app store or is it just that there would need to be some tweaking to get a briefcase app approved? curious bc while i'm initially planning to distribute my app outside the app store i still had at least some vague idea that eventually with some cleanup of entitlements etc. i could offer submit it.

To the best of my knowledge, there's nothing inherently preventing App Store submission of a Briefcase app. All I'm saying is we (a) don't currently test for macOS App Store-compatible configurations, and (b) don't have a briefcase publish macOS --to-app-store command.

If the process is anything like iOS App Store submission, there's a bunch of stuff that needs to be done around setting signing certificates, and validations that are performed during the archival/submission process. I have no idea if the current Briefcase app will fall foul of these validations - there's an outside chance that the com.apple.security.cs.allow-unsigned-executable-memory entitlement that is installed by default might prevented App Store submission. This option was required to allow use of ctypes/FFI in the app (something Toga uses extensively); I'm not 100% certain if that entitlement is still required, or if that option is a problem for the App Store (it's the sort of thing that I could easily see as being prohibited).

Beyond that, any restrictions are much more likely to be specific to the individual app - e.g., attempting to access file system locations that are prohibited by the App Store Sandbox.

We're definitely interested in filling this gap though - "one click" publish-to-App-Store is definitely in Briefcase's long term roadmap, whatever the App Store involved - even if that's just a matter of documenting how to submit to the Mac App Store in the same way as we do for the iOS App Store.

michelcrypt4d4mus commented 3 weeks ago

ok that's what i was expecting - that it wasn't inherently setup for that given the very onerous restrictions around app store apps but that w/some banging around i could hope to sort it out.

just doing some experimenting in the last hour or so around sandboxing my briefcase app (bc that's a necessary step) and already ran into some stuff that has to be done via customization briefcase doesn't support that is relevant to this enhancement request... specifically my app tried to read the system-wide /etc/apache2/mime.types file and sandboxing stopped said "nope".

the answer was to add an entry to the entitlements but it has to be an array. briefcase docs say only strings and bools are allowed.

         <key>com.apple.security.temporary-exception.files.absolute-path.read-only</key>
        <array>
                <string>/private/etc/apache2/mime.types</string>
        </array>

freakboy3742 commented 3 weeks ago

the answer was to add an entry to the entitlements but it has to be an array. briefcase docs say only strings and bools are allowed.

Looking at the implementation, it looks like lists and dictionaries should also be supported (although the ultimate value types in any list/dict will need to be strings and bools). Try it and see if it works; and if it does, a documentation update may be in order.

michelcrypt4d4mus commented 3 weeks ago

i tested it and yeah you're right, lists work fine. here's a documentation change if you want it.

Looking at the implementation,

where exactly is that implementation? i tried to check myself in case it was just a documentation error but couldn't find anything - only code i found about entitlements was this line

michelcrypt4d4mus commented 3 weeks ago

also it looks like the SKIP_INSTALL build setting is unnecessary in my situation.

freakboy3742 commented 3 weeks ago

i tested it and yeah you're right, lists work fine. here's a documentation change if you want it.

Dict values are also good; other than that, that change looks good. Feel free to submit a PR (including the changenote) and I'll merge it.

Looking at the implementation,

where exactly is that implementation? i tried to check myself in case it was just a documentation error but couldn't find anything - only code i found about entitlements was this line

It's being handled by the jinja filter handler for plist values.

michelcrypt4d4mus commented 3 weeks ago

there's an outside chance that the com.apple.security.cs.allow-unsigned-executable-memory entitlement that is installed by default might prevented App Store submission. This option was required to allow use of ctypes/FFI in the app (something Toga uses extensively); I'm not 100% certain if that entitlement is still required, or if that option is a problem for the App Store (it's the sort of thing that I could easily see as being prohibited).

i have been concerned from the get go about this exact entitlement but thankfully it seems like I was able to remove it from my application without anything breaking.

I'm not sure if my app ever uses ctypes directly but I know for sure it does directly make use of some Python / ObjC integrations. It's not a GUI app but at a minimum i've been using the packages listed below without issue despite the lack of the com.apple.security.cs.allow-unsigned-executable-memoryentitlement (so far, anyways).

from AppKit import NSImage
from Foundation import NSDate, NSObject, NSURL
from PyObjCTools import AppHelper

michelcrypt4d4mus commented 3 weeks ago

FWIW my app also seems to work fine when i disable the other dangerous entitlement that briefcase turns on by default (com.apple.security.cs.disable-library-validation)... app store might be closer to reality than i was assuming.

freakboy3742 commented 3 weeks ago

I'm not sure if my app ever uses ctypes directly but I know for sure it does directly make use of some Python / ObjC integrations.

Based on your code sample, it looks like you're using PyObjC, which AFAIK doesn't use ctypes; it's directly compiled against the native APIs. That probably means you're safe from ctypes issues.

michelcrypt4d4mus commented 3 weeks ago

ran into some stuff around ctypes and entitlements relevant to the discussion in the last few posts but not really relevant to the original feature request so to avoid cluttering up this thread i started a discussion about it: https://github.com/beeware/briefcase/discussions/1868

beeware / briefcase