Updated app bundle missing the MacOS executable (app "damaged or incomplete")

[iamcalledrob]

Description of the problem

We have a user reporting that our app suddenly stopped working on their machine after an update.

Attempting to launch the app reports You can't open the application “Blip” because it may be damaged or incomplete. Thousands of other users have updated without a problem. They were updating from v1.0.5 to v1.0.6 of the app.

Inspection of the app bundle sent by the user shows an intact app bundle for v1.0.5, but missing the executable in the Contents/MacOS folder. The folder exists, there's just nothing in it.

Debugging clues:

1. This user is also reporting that they're now also unable to connect to my updates server. This server hosts both the appcast xml and the update files. They are getting "Safari can't connect to the server", so it's unclear if it's a DNS or firewall issue. We're using a bucket on Backblaze b2 -- nothing fancy.
2. Due to a bug in one of our release scripts, the most recent appcast (release v1.0.6) was generated without the prior release (v1.0.5) in the work folder. An appcast and delta updates were generated from the previous version (v1.0.4)

Questions:

1. Are there any circumstances where an unreliable connection to the update server could lead to a corrupt app bundle?
2. Could there be a bug when applying delta updates when a prior release is missing from the work folder when running generate_appcast?
3. Has anyone encountered this sort of issue and discovered a root cause, in case it's actually not Sparkle at fault here? I've seen some threads about 3rd party "mac cleaner" or "space saver" apps potentially futzing with things? (Though the user reports nothing like that installed)

This is a really serious issue for us, so I'd love some help debuging what's happened here -- or at least make an educated guess :)

Do you use Sandboxing in your app?

No, the app is non-sandboxed

Version of Sparkle.framework in the latest version of your app

2.4.2

Version of Sparkle.framework in the old version of app that your users have (or N/A)

2.4.2

Sparkle's output from Console.app

This didn't reproduce on my machine, so I don't have logs.

Steps to reproduce the behavior

Corrupt app bundle from user's machine: Blip.zip

appcast.xml (at time of writing)

``` Wed, 13 Dec 2023 09:08:34 -0800 beta 20231213083601 1.0.6 12.0 Wed, 22 Nov 2023 14:28:50 -0800 beta 20231122140841 1.0.4 12.0 Mon, 20 Nov 2023 14:54:07 -0800 beta 20231120143754 1.0.3 12.0 ```

[zorgiepoo]

Just yesterday I was investigating any potential corruption issues on our end and one issue I found is https://github.com/sparkle-project/Sparkle/pull/2479 which I'm fixing. This specific issue is most likely trigger as the system is about to shut down and an update is waiting to be installed (most likely when automatic downloading of updates is opted into but still possible to encounter with it off). The suspected code was introduced in 2016 and is likely rare and not easy to reproduce.

This is the first report of a missing macOS executable issue here.

As far as ruling out other aspects:

• Updates are installed/swapped atomically with the filesystem so it's not normally possible for a 'corrupt' install to occur
• With HTTPS it's not possible for the download to be "corrupt" and not what the server served. If you're using EdDSA signing, Sparkle also ensures it downloads the correct archive data.
• Delta updates have full verification in place for the whole app bundle to ensure the update is applied properly; generate_appcast these days has verification in place to ensure the deltas it creates can be applied successfully
• Shallow Apple code sign verification is also done after extraction which would catch a missing main executable (after applying deltas too).
• It is technically possible for a bad actor to damage the bundle which may reside in a Caches directory. This could be a cleaner app or incidental like in #2479

I will close this after #2478 lands as that is the only lead I have here. Sorry about any trouble here :\

[iamcalledrob]

Thank you @zorgiepoo - what a fantastic follow-up!

We'll keep an eye on this and bump our toolchain to use a newer version of Sparkle once there's a new release with #2479 included.

For future, is there any additional logging we should be adding to help pinpoint the bug this if it happens again?

[zorgiepoo]

I just put out a minor update with this change.

As for additional logging, I am not sure. But I would suspect it is something that modifies the bundle after Sparkle extracts/validates the bundle without any issue, especially if codesign --verify (without --deep) can catch the issue. Gatekeeper should also protect the user from running a damaged update as you saw here.