Self patching Mac application

[colincornaby]

This adds self patching to the Mac application. Servers will need to offer a Mac specific manifest, and encode the Mac client app with the bundle flag. The Mac client should be encoded in a tar file before it is gzipped. In the current implementation, the root of the inside of the bundle is the top level of the tar file.

The hash of the Mac client should be derived from the executable inside. (Note: this differs from the current implementation of https://github.com/Hoikas/UruManifest/pull/12.) The executable should be code signed. Because the executable is code signed, it contains hashes as part of its loader flags that validate the entire bundle. This guarantees a unique executable for any version of the bundle - even if only resources have changed.

On macOS, CFBundle is used to decode the bundle and find the executable within the bundle. The name of the executable is encoded in the info.plist of the bundle, but non Apple platforms could either include Swift or CoreFoundation to decode the bundle, or make a reasonable guess. This could be useful for encoding manifests on non-Mac platforms.

Ventura has protection against apps being able to modify installed applications. I noticed this while implementing this feature, but am yet to test the bounds of this security or research further. It seems that macOS will permit us to install a new application on top of our existing application. But we may be prevented from upgrading any other bundle on the system. There may be ways around this using code signing.

This change also disables use of the Windows manifest files meaning after this change the Mac client will no longer work with Windows servers. A reasonable change could be an "emulate Windows" launch flag that causes the Mac client to go back to it's previous behavior of using the Windows manifests and filtering out executables.

Draft notes:

• Getting this in front of people for early review.
• Error handling likely needs to be improved.
• Not all the patcher flags from Windows are implemented yet.

[dpogue]

My first question, which is sort of a meta question: Is there a reason to do this in plClient rather than plUruLauncher, to match Windows?

On Windows, plUruLauncher handles the patching (and self-patching of just the launcher) and then launches plClient.

If we can do it all as a single binary, that seems better for end users, but I'm not sure what the original pros/cons were of having plUruLauncher be separate.

[colincornaby]

I'm not sure. On the Mac it would be more complicated in since there are no start menu links - so we'd have to stash plClient someplace it wouldn't confuse the user.

I don't know that the launcher really simplifies things easier. Self patching is a pain point but the launcher has to self patch itself too.

[colincornaby]

Here's a post talking about the new app update restrictions in Ventura: https://lapcatsoftware.com/articles/AppManagement.html

It sounds like - as long as updates continue to be signed by the same signing organization - updates should work. But apps that are signed by one signing entity are not allowed to update apps signed by other entities. This could cause problems if an Uru server suddenly changed who was signing the application.

[colincornaby]

Reading more - the only thing I'm concerned about is the insinuation that feature should only apply to applications that attempt to alter the bundle of another application - and not replacing the entire bundle. Odd.

[dpogue]

• Because the bundle updating logic is macOS-specific anyway, could the archive handling be done using a macOS system library/framework/tool instead of pulling in our own copy of libarchive?

No, because while macOS includes libarchive as a system library, it is not part of their public API and they don't provide headers for it and you're not allowed to link against it.

We talked initially about using the Apple-specific XAR archive format, but that posed issues for generating the bundles on Linux/Windows machines (where the servers are probably running).

Apple has a new AppleArchive format that they're wanting people to use, but it's only supported on macOS 11.0 and newer.

[dgelessus]

I was thinking of the Apple-specific frameworks and not libarchive, but the Apple Archive framework is only supported since macOS 11 (and is apparently Swift-only) and the Compression framework only does single files and not archives. So none of those are an option.

I suppose you could call Archive Utility or the command-line tar tool to unpack the .tar.gz, but that's probably a lot more work and not worth it just to avoid the libarchive dependency.

[colincornaby]

On signing specifically - unsigned applications won't run on macOS anymore without some serious tinkering. On Apple Silicon I believe it requires a firmware level override.

How everyone signs is still a complicated question with no good answer yet. But it's very likely everything distributed for macOS will need to be signed. Unfortunately - if not done right - that could cause some servers to not roll out a Mac client. If H'uru could control app builds and distribution for all shards (and we figure out signing) we could allow other shards to distribute our signed client.

[dgelessus]

Hmm, that requirement is only true for native ARM code on Apple Silicon machines, I think. I have a couple of x86_64 applications that I can run on my M1 MBP even though they are (according to codesign -dv) "not signed at all".

In any case, I think ad-hoc signatures will be enough for testing purposes. They don't require any kind of code signing identity, so anybody can ad-hoc sign an app using the standard Apple dev tools. (I think this happens automatically for ARM code, which is how you can run self-compiled ARM binaries even if you never explicitly sign them.) Although ad-hoc signatures are only valid on the machine that created them, you can bypass that check with just right-click > Open on an ad-hoc signed app. I think ad-hoc signed binaries also contain all the resource hashes that a real signed binary would, so they should be enough to make the hash of the main binary representative for the entire bundle.

TLDR: if you ad-hoc-sign your client, everything should be fine.

[colincornaby]

I don't know that I'd encourage ad hoc signing because of notarization - which is the other requirement looming over the Mac client. I don't think notarization works with ad hoc app.

Of course notarization can also be bypassed by lowering Gatekeeper settings and right click -> opening the app. But I don't know if that's a good experience. I'm also unsure of our support path for Intel binaries. Apple may stop producing tools and may stop producing Rosetta (at least for Cocoa apps) at some point in the future. So I weigh the experience with an Apple Silicon binary a lot more strongly than an Intel binary.

(How we handle dropping support for an architecture or OS revision may be it's own question with the self patcher.)

Regardless - distributing unsigned should be an edge case, and it would only be an issue if somehow a release was done without the executable changing at all. Which is possible, but an edge case within an edge case.

[dgelessus]

To clarify, I was talking about ad-hoc signing as a better alternative than completely unsigned binaries, for cases where real signing isn't an option (mainly for people who don't pay Apple 100 $ a year). Of course, if you're able to, you should build a properly signed binary and notarize it. I just wanted to point out that if you don't have that ability, you can still produce a client that works with the bundle patching logic and can be run by normal people without too much hassle.

I don't think we need to worry about dropping support for x86_64 for now. The latest version of macOS still supports it, and as long as that's the case, the dev tools will be able to target it too, I assume.

[dpogue]

That way, we aren't requiring folks to pay the Apple tax.

FWIW, I'm pretty sure you can do local code signing with a free Apple Developer account, you only need to paid if you want to set up a team with multiple people having access to the signing certificates (and possibly also for notarization?)

That said, trying to do code signing on a build system where you're not logged in to the Xcode GUI is a pain, and I have spent literally 12 years of my life fighting that battle for CI builds of iOS apps at work.

[colincornaby]

That way, we aren't requiring folks to pay the Apple tax.

FWIW, I'm pretty sure you can do local code signing with a free Apple Developer account, you only need to paid if you want to set up a team with multiple people having access to the signing certificates (and possibly also for notarization?)

That said, trying to do code signing on a build system where you're not logged in to the Xcode GUI is a pain, and I have spent literally 12 years of my life fighting that battle for CI builds of iOS apps at work.

The free program won't let you sign or notarize - only run locally (or technically self sign for local runs.) The $99 individual program will let you sign, or the $299 group program.

At a PR conversation level - I still remain unconvinced that groups would be able to realistically avoid code signing. Code signing and notarization are both required on Apple Silicon. I think Intel only builds being grandfathered in is a loophole that will only work for a limited period of time. Rosetta will eventually go away, our ability to build for Intel will go away over the long term, and I wouldn't really want to encourage Intel only builds.

By the same token - developer/self sign/adhoc builds cannot be distributed to Apple Silicon users because they cannot be notarized.

That way, we aren't requiring folks to pay the Apple tax.

I don't think we're doing anything. Unfortunately it's Apple requiring people to pay the Apple tax. Whatever our feeling are, Apple at an operating system level is increasingly forcing the matter.

We can't give away the signing keys - but one service we could provide is distribution of H'uru signed clients.

[Hoikas]

Apple will do as Apple will do. I remain unconvinced that we should acquiesce to their money and/or time grab. I understand that Mac on ARM will require paying Bill GatesTim Cook, but I don't see why we should simply capitulate to that requirement when there is a suitable solution for this problem in the code already in the repo. Not to say we shouldn't encourage signed builds nor should we not provide signed builds (I think it would be great if we did), I think that providing something that works easily and inexpensively out of the box is important. Intel Macs aren't going away any time soon - I know that Apple likes for people to throw their computer away every 4 years or so, but, realistically, anything mid-range or better from the last decade is still perfectly serviceable (or better) hardware. Just because Apple will stop caring doesn't mean we should, IMO.

[dpogue]

The free program won't let you sign or notarize - only run locally (or technically self sign for local runs.)

Ahh right, and I guess when we're talking about patching it's unlikely being are doing local builds. So builds meant to be distributed via the patcher would always need to be signed/notarized, yeah.

[dgelessus]

By the same token - developer/self sign/adhoc builds cannot be distributed to Apple Silicon users because they cannot be notarized.

Notarization isn't specific to ARM. It's been "required" since macOS 10.15, but it's relatively easy for the user to bypass that. AFAIK, this behavior is always the same, regardless of the architecture of the binary and the host.

An unnotarized ad-hoc signed build definitely can be distributed, even if it's native ARM running on Apple Silicon. AFAIK, the only hurdle is that the end user needs a couple of extra clicks the first time they run the app (the user does not need to run scary commands or disable important system security features). As mentioned above, I think that's acceptable for testing purposes.

Because ad-hoc signing is done with the standard dev tools and doesn't require a code signing certificate or other special permission, it should be okay to assume that anybody can do ad-hoc signing, so we shouldn't need other workarounds like force-embedding the build timestamp. (Please also keep in mind that not every client build will go onto a file server. 🙂)

[colincornaby]

Apple will do as Apple will do. I remain unconvinced that we should acquiesce to their money and/or time grab. I understand that Mac on ARM will require paying ~Bill Gates~Tim Cook, but I don't see why we should simply capitulate to that requirement when there is a suitable solution for this problem in the code already in the repo. Not to say we shouldn't encourage signed builds nor should we not provide signed builds (I think it would be great if we did), I think that providing something that works easily and inexpensively out of the box is important. Intel Macs aren't going away any time soon - I know that Apple likes for people to throw their computer away every 4 years or so, but, realistically, anything mid-range or better from the last decade is still perfectly serviceable (or better) hardware. Just because Apple will stop caring doesn't mean we should, IMO.

I think I'd like to see this in a place where solutions are more productive for working with the ecosystem that exists. Providing signed H'uru builds for other servers doesn't seem infeasible. UruManifest already pulls our GitHub builds - it wouldn't be a stretch for it to pull signed releases that get sent downstream to other shards.

Intel support is going to be beyond our control. Eventually Apple will stop supplying Intel SDKs, we won't be able to run the old ones on current hardware/operating systems, and the old SDKs will drop off of Github Actions. It may be some time (probably 3-4 years from now?) But it's going to be out of our control. Same thing happened with PowerPC. You couldn't actually build for PowerPC anymore unless you kept specific hardware/software configs around to do it. It's not a stance on the longevity of old hardware. It's just me assuming that there will come a day when we cannot build against the Intel SDK. Apple does not support building against old SDKs long term like Microsoft does and actively blocks it in their software.

To be frank - there is already considerable overhead in our support range. I have an Nvidia Macbook Pro that has it's GPU power management controller wedged into place with a rubber pad because it's become partially desoldered - and it's my only Nvidia Mac. Uru is an older game, and older hardware may be capable. But the actual overhead of supporting older Macs in development is and will be a problem. I have to maintain the Macs - and I have to have an OS install with the matching SDK for each OS version we support (which the Metal debugger gets really picky about.) And because Apple doesn't distribute free standing GPU driver updates, when we support a macOS version, we support all the GPU driver bugs on that macOS version. Which has been a particular problem with the Nvidia and Intel GPU drivers.

I guess what I'm saying is I agree in principle supporting older hardware is great - but the technical overhead of doing that on macOS is very different than Windows.

[colincornaby]

By the same token - developer/self sign/adhoc builds cannot be distributed to Apple Silicon users because they cannot be notarized.

Notarization isn't specific to ARM. It's been "required" since macOS 10.15, but it's relatively easy for the user to bypass that. AFAIK, this behavior is always the same, regardless of the architecture of the binary and the host.

An unnotarized ad-hoc signed build definitely can be distributed, even if it's native ARM running on Apple Silicon. AFAIK, the only hurdle is that the end user needs a couple of extra clicks the first time they run the app (the user does not need to run scary commands or disable important system security features). As mentioned above, I think that's acceptable for testing purposes.

Because ad-hoc signing is done with the standard dev tools and doesn't require a code signing certificate or other special permission, it should be okay to assume that anybody can do ad-hoc signing, so we shouldn't need other workarounds like force-embedding the build timestamp. (Please also keep in mind that not every client build will go onto a file server. 🙂)

There is one angle to this I probably still need to do more digging on, but it could be relevant here.

Ventura added the requirement that when an application modifies another application - the code signing team needs to match. In testing, it did not seem like my developer signed build trying to do any sort of self patching. It's not clear what an unsigned build would do or if an unsigned build would be allowed to self patch (my guess is no - but thats just a guess.)

I need to contact Apple DTS to get more clarity on the restrictions. My big concern has been if a server changed its code signing - self patching may begin to fail. But it might cause unsigned applications to not be able to patch at all.

(The app modification policy can be changed in security - but it's another toggle for the user to go through... And the OS will just keep blocking self patching quietly until the user toggles that setting. The user will only be notified on the first attempt.)

[dgelessus]

Ventura added the requirement that when an application modifies another application - the code signing team needs to match.

I read about this a few days ago, but unfortunately can't find the link anymore... but if I remember correctly, this restriction is only relevant if you modify individual files in an existing app bundle. If you replace the entire bundle with another one (which your patching code does here I think), then the teams don't need to match.

[Deledrius]

That makes sense from an internal-consistency point of view.

[colincornaby]

I read about this a few days ago, but unfortunately can't find the link anymore... but if I remember correctly, this restriction is only relevant if you modify individual files in an existing app bundle. If you replace the entire bundle with another one (which your patching code does here I think), then the teams don't need to match.

That is what I thought as well, but it's getting triggered on full bundle replacements that do not access the internals of the bundles, or do partial replacements. This PR triggers it even though it does a swap at a file system level.

The documentation is kind of ambiguous - but the WWDC sessions specifically mentions self update systems. Sparkle also had a thread on this issue where they determined that they should be ok because they inherit the parent apps signing. It's one reason I want to follow up with Apple. Most the security oriented documentation seems concerned about replacement of bundle internals. But there is some documentation out there implying this also affects self update systems.

[colincornaby]

I've reached out to Apple to inquire why we're seeing the App Management security feature block self updating in unsigned situations (where at least one bundle is unsigned) and if that is intended behavior. It may be a bit, I'll update when I hear back.

[colincornaby]

I'm going to resume working on this. I haven't seen any response from Apple yet - and I still need to investigate what is going on with self patching when signing changes on one of my Macs (prompting an application modification block.) But I've built other standalone samples that don't seem to cause blocking issues.

[colincornaby]

I've resolved the questions about the App Modification security block - and have moved the Mac bundle code into the Mac client itself. I'm going to promote this to a full PR.

Note that UruManifest behavior still does not match the current implementation. I'm testing with a hand modified manifest - I'll need to update UruManifest separately.

[colincornaby]

As per the previous discussions, the overall approach seems good to me. The review comments are mostly minor things and no major design issues. Two remaining questions:

• Do we care about what happens when non-macOS patchers try to download a bundle?
• To clarify: because the bundle patching logic is implemented as part of the client self-patching code, this means that there is no support for bundles other than the main client application - correct?

My assumption is that bundles should only be specific to Mac manifests.

Originally the bundle approach was more open ended and was basically a folder diff. These changes make the assumption that a bundle has an executable that can be used for the size check and hash. That not only constrains the usefulness of the bundle flag - but it even excludes non executable bundles on macOS.

I would have preferred a more open approach - but this approach is more reliable with a simpler implementation. So I can't complain too much.

Introducing a code path and a manifest flag for macOS may also not be the worst thing. There are specific callbacks for things like the Visual Studio runtime. (Is that a flag in the manifest as well?)

I did also consider if we'd ever need to support patching a macOS bundle without an executable inside. I'd assume we'd have little reason to ship a bundle as a sidecar to the Uru app.

I'm open to feedback - but I think we're somewhat constrained by the existing patcher implementation being so file focused. It would have been nice if the manifest was open ended enough to group files so that if any file in the group failed checksum comparison all files in the group are replaced.

[colincornaby]

I'm working through the error cases that @dgelessus pointed out. One important thing on macOS that I should raise (that could be a source of errors.)

Uru expects its directory to be writable by the current user. I should be noted that a non admin user does not have write access to the Applications directory.

Other games like World of Warcraft handle this by making their directories world writable - even if that's not consistent with other Applications. My thinking is that Uru should handle this the same. That means Uru will need an installer on macOS that can properly set permissions on its directory.

The way this is supposed to be handled is privilege escalation through a secondary process - but because Uru has a streaming model that streams new assets at runtime, this would be difficult to do.

I'm not sure if anyone has any thoughts on this. This would make drag and drop installation to Applications basically unsupported.

[dgelessus]

That's a good point, I wasn't even thinking about that specifically. I wonder how other self-updating macOS applications handle this - you could check Sparkle (the de facto standard macOS self-updater framework) or ShipIt (the thing used by Electron-based applications, I think). I don't remember ever running into permission issues with applications updating themselves in the Applications folder.

That said, if for some reason our patcher can't work properly in the Applications folder, we could tell users to put the Uru client somewhere in their user folder instead. Not an optimal solution obviously, but we already have a similar thing on Windows, where we always tell people to not install Uru into Program Files.

[colincornaby]

Applications based on Sparkle only update once at launch (or whenever the user delays to) - so they can request admin privs and then do their thing. You can't permanently hold admin privs, but they're only need for a short time to do a one time update.

Uru is a little different because it patches as you play the game. So unless you request admin privs every time the user goes to a new region and needs new files, you won't be able to patch.

Again - WoW has the same problem because it dynamically downloads content as you move around. And they won an Apple Design Award. So I think we're ok to have an installer that just makes the Uru dir world writable.

If someone just downloads Uru on its own for development purposes then they're on their own. But that should be more rare.

[Hoikas]

Not an optimal solution obviously, but we already have a similar thing on Windows, where we always tell people to not install Uru into Program Files.

Right, this sounds like basically the same situation as on Windows. I always recommend shard operators have an installer that does the proper directory permissions magic because we, in truth, have no idea what kind of environment we're being installed into. If other Mac games are doing installers to set directory permissions, that really lines up well with what the rest of the Uru ecosystem should be doing.

[dgelessus]

Applications based on Sparkle only update once at launch (or whenever the user delays to) - so they can request admin privs and then do their thing.

The Sparkle-based apps I use (e. g. BBEdit, VLC) have never asked me for admin privileges to self-update, even though I installed them by the usual "drag from disk image to Applications" method and never touched the permissions of anything. I see that the app bundles and all files inside are owned by my user and not "System". In that case there should be no issues with Uru updating its own files, right?

[colincornaby]

The Sparkle-based apps I use (e. g. BBEdit, VLC) have never asked me for admin privileges to self-update, even though I installed them by the usual "drag from disk image to Applications" method and never touched the permissions of anything. I see that the app bundles and all files inside are owned by my user and not "System". In that case there should be no issues with Uru updating its own files, right?

You may not see this because if you're using an admin account on your Mac - you will have write permissions into the Applications directory. This is the common case for single user Macs.

Secondary accounts with the "Standard" permissions cannot freely write to the Applications directory. I looked over the Sparkle code while preparing this branch - Sparkle will prompt for admin privs escalation and spin up a secondary process that inherits those privileges to do the patch where permissions do not allow.

This also means that a Standard user cannot install applications either - although some installers allow a Standard user to install to ~/Applications instead.

Another sanity check could be that Plasma at launch does a permissions check on its own directory to make sure that it can write to the directory. If there was a launch time check that would give us more assurance that we're not going to run into issues later with not being able to write.

[dgelessus]

You may not see this because if you're using an admin account on your Mac - you will have write permissions into the Applications directory. This is the common case for single user Macs.

Ah right, that would make sense. In that case, I don't think we have anything to worry about, because most users will have write access to Applications without any tricks.

In the rare case that the user is not an admin account and then tries to install Uru into Applications, they will get an authentication dialog at install time, which should make it easy to guess the problem when the self-updater later fails with permission errors. In those cases, I think we should tell people to simply not do that, rather than adding an installer just for adjusting the permissions.

Another sanity check could be that Plasma at launch does a permissions check on its own directory to make sure that it can write to the directory. If there was a launch time check that would give us more assurance that we're not going to run into issues later with not being able to write.

That would be a good idea (regardless of how we decide to handle the permission stuff).

[colincornaby]

Do we want to maintain some capacity of loading from the Windows manifest like the Mac client has today? It could be useful for connecting to a Windows only instance for diagnostics.

Technically this change would leave the Mac client unable to do any patching from our current instances. I know we'll eventually ship a Mac client - but it would be nice to have a flag that at least gives us some compatibility.

[colincornaby]

I did figure out why I encoded the tar file to not include the application bundle root folder. One ambiguity is that a tar file can contain multiple files. If a tar file was accidentally encoded with multiple files at the root - this could cause a lot of mess. By assuming the tar contains the contents of the bundle - and not the bundle folder itself - I can remove that ambiguity.

It might be possible to have the tar contain the application bundle and not just the contents of the bundle. But I'd have to do more introspective on the tar file and look for the bundle specifically - and not just blindly decompress the entire tar file.

[colincornaby]

I've been busy - but I've been quietly continuing to look at this PR. Where I'm currently at:

• I'm still pretty happy with the general direction of this PR - even if it is not terribly elegant and requires modifications to the patch system. It is unfortunate that Uru's manifest system is file by file - but it is what it is.
• I'm still fairly certain that the initial install will need to be done with an Installer. I've found no good solution for the permissions issue - which ripples into the in game patching. -However-
• I have been working on an installer - and the installer supports relocation. So if you want to install it to a custom location that is not the global applications folder, you can.
• The catch is it would be nice if Uru was smarter about permissions. If you install Uru into a personal directory, it should inherit your personal permissions. But that requires the patcher to be smarter.
• I need to go revisit the Python manifest generator code that is in a companion pull request that is out of sync.
• I'm not sure if these manifest changes need any documentation. If the Mac client is upstreamed to MOULA - there will be some Mac specificities for manifest creation.

[colincornaby]

I'll take a look at the use of NSFileManager. The way I'd probably evaluate that is how deeply the usage is integrated with Cocoa. (I.E integrated with NSBundle or Cocoa error reporting paths.)

[colincornaby]

Changes are made and tested against local Docker server.