search

sparkle-project / Sparkle

A software update framework for macOS
https://sparkle-project.org
Other
7.41k stars 1.05k forks source link

zip files generated from Xcode Cloud cannot be streamed during extraction #2544

Closed sktusing closed 1 month ago

sktusing commented 5 months ago

Description of the problem

Using generate_appcast with a .app.zip generated from Xcode Cloud results in an error:

sktusing@SKT-MBP Sparkle-2.6.0 % ./bin/generate_appcast /Users/sktusing/Desktop/Vor_Release 
ditto: Vor.app/Contents/Frameworks/DeckLink.framework/DeckLink: No such file or directory
ditto: Couldn't read pkzip signature.
ditto: Couldn't read pkzip signature.
ditto: Couldn't read pkzip signature.
ditto: Couldn't read pkzip signature.
ditto: Couldn't read pkzip signature.

Running generate_appcast a second time results in:

sktusing@SKT-MBP Sparkle-2.6.0 % ./bin/generate_appcast /Users/sktusing/Desktop/Vor_Release 
Could not unarchive /Users/sktusing/Desktop/Vor_Release/Vor.app.zip Error Domain=NSCocoaErrorDomain Code=516 "“Vor.app.zip” couldn’t be copied to “762209f0516ccdac78f286882388fccbe8cfbae5c16c22f1946db52ce85f7926.tmp” because an item with the same name already exists." UserInfo={NSSourceFilePathErrorKey=/Users/sktusing/Desktop/Vor_Release/Vor.app.zip, NSUserStringVariant=(
    Copy
), NSDestinationFilePath=/Users/sktusing/Library/Caches/Sparkle_generate_appcast/762209f0516ccdac78f286882388fccbe8cfbae5c16c22f1946db52ce85f7926.tmp/Vor.app.zip, NSFilePath=/Users/sktusing/Desktop/Vor_Release/Vor.app.zip, NSUnderlyingError=0x6000012e4d20 {Error Domain=NSPOSIXErrorDomain Code=17 "File exists"}}
Error generating appcast from directory /Users/sktusing/Desktop/Vor_Release 
 Error Domain=SUSparkleErrorDomain Code=1001 "No usable archives found in /Users/sktusing/Desktop/Vor_Release" UserInfo={NSLocalizedDescription=No usable archives found in /Users/sktusing/Desktop/Vor_Release}

Running sktusing@SKT-MBP ~ % zipinfo /Users/sktusing/Desktop/Vor_Release/Vor.app.zip shows that Decklink directory exists:

lrwxrwxr-x  2.0 unx       25 bX stor 24-Apr-21 14:21 Vor.app/Contents/Frameworks/DeckLink.framework/DeckLink

Unzipping, then zipping the app causes generate_appcast to function.

This is new with us migrating to Xcode Cloud for notarizing the application.

Do you use Sandboxing in your app?

No

Version of Sparkle.framework in the latest version of your app

2.6.0

Version of Sparkle.framework in the old version of app that your users have (or N/A)

N/A

Sparkle's output from Console.app

default 10:18:31.749037-0400    generate_appcast    Not internal release, disabling SIRL
default 10:18:31.749078-0400    generate_appcast    Enabling System Keychain Always due to platform
default 10:18:31.749119-0400    generate_appcast    [0x600003acc000] activating connection: mach=true listener=false peer=false name=com.apple.securityd.xpc
default 10:18:31.749536-0400    secd    Entitlement com.apple.application-identifier=org.sparkle-project.Sparkle.generate-appcast is ignored because of invalid application signature or incorrect provisioning profile
default 10:18:31.750303-0400    generate_appcast    [0x600003ac0000] activating connection: mach=true listener=false peer=false name=com.apple.cfprefsd.daemon
default 10:18:31.754100-0400    generate_appcast    UNIX error exception: 17
default 10:18:31.759236-0400    generate_appcast    UNIX error exception: 17
default 10:18:31.760170-0400    generate_appcast    UNIX error exception: 17
default 10:18:31.762017-0400    generate_appcast    UNIX error exception: 17
default 10:18:31.763014-0400    generate_appcast    UNIX error exception: 17
default 10:18:31.764159-0400    generate_appcast    UNIX error exception: 17
default 10:18:31.802500-0400    generate_appcast    [0x600003ad8000] activating connection: mach=true listener=false peer=false name=com.apple.cfprefsd.agent
default 10:18:31.813110-0400    generate_appcast    [0x14a205480] activating connection: mach=true listener=false peer=false name=com.apple.lsd.mapdb
default 10:18:31.814565-0400    generate_appcast    [0x600003ac42d0] activating connection: mach=true listener=false peer=false name=com.apple.distributed_notifications@Uv3
default 10:18:32.905052-0400    generate_appcast    Not internal release, disabling SIRL
default 10:18:32.905095-0400    generate_appcast    Enabling System Keychain Always due to platform
default 10:18:32.905138-0400    generate_appcast    [0x600000948000] activating connection: mach=true listener=false peer=false name=com.apple.securityd.xpc
default 10:18:32.905350-0400    secd    Entitlement com.apple.application-identifier=org.sparkle-project.Sparkle.generate-appcast is ignored because of invalid application signature or incorrect provisioning profile
default 10:18:32.906096-0400    generate_appcast    [0x6000009480f0] activating connection: mach=true listener=false peer=false name=com.apple.cfprefsd.daemon
default 10:18:32.909018-0400    generate_appcast    UNIX error exception: 17
default 10:18:32.910882-0400    generate_appcast    UNIX error exception: 17
default 10:18:32.912028-0400    generate_appcast    UNIX error exception: 17
default 10:18:32.913280-0400    generate_appcast    UNIX error exception: 17
default 10:18:32.914386-0400    generate_appcast    UNIX error exception: 17
default 10:18:32.915625-0400    generate_appcast    UNIX error exception: 17
default 10:18:32.952601-0400    generate_appcast    Extracting using '/usr/bin/ditto' '-x' '-k' '-' < '/Users/[user]/Library/Caches/Sparkle_generate_appcast/762209f0516ccdac78f286882388fccbe8cfbae5c16c22f1946db52ce85f7926.tmp/Vor.app.zip' '/Users/[user]/Library/Caches/Sparkle_generate_appcast/762209f0516ccdac78f286882388fccbe8cfbae5c16c22f1946db52ce85f7926.tmp'

Steps to reproduce the behavior

  1. Download zipped application from Xcode Cloud
  2. Sign .app.zip with generate_appcast
  3. generate_appcast errors and fails
zorgiepoo commented 5 months ago

Can you try unzipping the archive with the following command with Terminal:

ditto -x -k Vor.app.zip .

If it fails, theres's a chance the zip file is not that good, and you wouldn't want to distribute a zipped app that will be prone to fail extracting with one of the system unarchivers. Which macOS version are you on, too?

As for the second error, try to remove the offending extracted cache generate_appcast generated, which I think is bad because it failed to originally extract the file properly the first time. As the error you provided indicates, that may be in ~/Library/Caches/Sparkle_generate_appcast/762209f0516ccdac78f286882388fccbe8cfbae5c16c22f1946db52ce85f7926.tmp

sktusing commented 5 months ago

Can you try unzipping the archive with the following command with Terminal:

The command does not fail, and the app is extracted.

Which macOS version are you on, too?

14.4.1 (23E224)

zorgiepoo commented 5 months ago

The last line here shows the exact (barring user name stripped out) command generate appcast uses

default 10:18:32.952601-0400    generate_appcast    Extracting using '/usr/bin/ditto' '-x' '-k' '-' < '/Users/[user]/Library/Caches/Sparkle_generate_appcast/762209f0516ccdac78f286882388fccbe8cfbae5c16c22f1946db52ce85f7926.tmp/Vor.app.zip' '/Users/[user]/Library/Caches/Sparkle_generate_appcast/762209f0516ccdac78f286882388fccbe8cfbae5c16c22f1946db52ce85f7926.tmp'

Does the issue always reproduce with that zip file? I don't think I can do much here without a debuggable repro case (like the zip file. I don't have Xcode cloud set up).

sktusing commented 5 months ago

Does the issue always reproduce with that zip file?

Yep, and others that have been generated since moving to Xcode Cloud for notarization. Happy to provide an un-signed zip file if you'd like.

zorgiepoo commented 5 months ago

Yeah it would be helpful providing a zip file. Feel free to email me (zorgiepoo at gmail dot com) if it's sensitive, otherwise link it here. There might be something wrong in our piping code in our unarchiver.

zorgiepoo commented 5 months ago

Received a couple zip files over email and can easily reproduce.

Unzipping, then zipping the app causes generate_appcast to function.

This is a workaround for now. I cannot reproduce trying several different ways of re-creating the zip file, so oddly it's specific to how Xcode Cloud creates zip files.

This issue seems to also be related to our NSPipe unarchiving code (shared between Sparkle and generate_appcast) https://github.com/sparkle-project/Sparkle/blob/0cff81740fec07c21e6206926ba9564a817ab46a/Autoupdate/SUPipedUnarchiver.m#L109

The ditto error is also pretty odd..

If generate_appcast does a signal(SIGPIPE, SIG_IGN) call like our unit tests and Autoupdate do, it won't abruptly halt, but generate_appcast will still emit an error.

Sparkle 1.x can't handle unarchiving these zip files as well, and I believe the code before then hasn't changed much since 2015. I guess I should next rule out if this is specific to generate_appcast or if other components are affected (generate_appcast has an extra symlink step).

zorgiepoo commented 4 months ago

Looking into this I found another unrelated bug I had to fix (https://github.com/sparkle-project/Sparkle/pull/2550), and which I'm still not done completely integrating in older releases.

This issue itself is that these zip files that Xcode Cloud generates cannot be streamed using a pipe during extraction correctly using any of the macOS built in tools (ditto, bsdtar). They will not extract these zip files and fail somewhere. That is not an issue in Sparkle. And this is not a new issue, just that no one has tried passing in these type of files before.

We could work around this by seeing if extraction fails due to SIGPIPE and re-trying without piping the file in to get extraction progress. But I am not sure if I want to support these zip files since we can't get any extraction progress and future support for failing to stream them is unknown (for example bsdtar doesn't even report an error, the files extracted are just incomplete/corrupted). Or we could completely remove in-progress streaming for zip files since it's not done in the most optimal way (via a pipe) but that doesn't sound great. Perhaps I should also more familiarize myself with the ZIP file and the file format to see what steps are needed to create zip files like these.

zorgiepoo commented 1 month ago

This bug is fixed in macOS 15 beta in ditto. However I also have a fix in #2616 for older OS's, which retries extraction without piping when piping fails (this may result in less fine-grained progress reporting, but may not be noticeable if the archive is small).