x/build/cmd/release: Go installer on macOS 12.2 beta (21D5039d) fails

alaroldai commented 2 years ago

What version of Go are you using (`go version`)?

Go Installer v1.17.6 (go1.17.6.darwin-arm64.pkg)

Does this issue reproduce with the latest release?

Yes (go1.18beta1.darwin-arm64.pkg)

What operating system and processor architecture are you using (`go env`)?

go env not available because the issue is in installing Go

MacOS 12.2 Beta (21D5039d), Mac mini (M1, 2020)

What did you do?

Downloaded Go installer from https://go.dev/dl/
Double-click to open with Installer.app
- MacOS refuses to open because the package is from an unidentified developer
Right-click and select "open" to bypass developer verification
- Installer fails and presents error message: "The operation couldn’t be completed. (com.apple.installer.pagecontroller error -1.)"

What did you expect to see?

Installer package should be signed, open without need for context-menu bypass or change to OS security settings
Installer should succeed

What did you see instead?

Installer package would not open without context-menu bypass
Installer failed

Other notes

I've since worked around this by installing via Homebrew, which worked without issues.

Reproduced the same issue with the following installer packages:

go1.18beta1.darwin-arm64.pkg
go1.18beta1.darwin-amd64.pkg
go1.17.6.darwin-amd64.pkg
go1.17.6.darwin-arm64.pkg

Also tried:

Extracting the package with pkgutil --expand: error Could not open package for expansion: go1.18beta1.darwin-arm64.pkg
Verifying signature with pkgutil --check-signature: error Could not open package: go1.18beta1.darwin-arm64.pkg
Extracting the package with xip --expand go1.18beta1.darwin-arm64.pkg: error xip: error: The archive “go1.18beta1.darwin-arm64.pkg” is damaged and can’t be expanded
Downloaded the installer via wget and tried all of the above, with the same results (except the signature-verification workaround wasn't needed because the file was downloaded via a terminal)

Verified signature manually using sha256sum and comparing against signature listed on https://go.dev/dl/

dmitshur commented 2 years ago

I tried on macOS 12.1 (21C52) and on macOS 11.6.2 (20G314), and this issue doesn't reproduce. The installer shows up as correctly signed with a Google certificate, and opens okay.

So this may be early signal of a regression in a future macOS release, a problem with beta macOS releases in general, or a problem with the macOS installation on the device you tested with. Are you able to add more information to help us determine if it's one of those possibilities? Thanks.

CC @golang/release.

cherrymui commented 2 years ago

I have an M1 laptop, which was running a previous version of 12.2 beta (21D5025-ish, I don't recall the exact number), and the installer runs fine. I just updated it to a newer beta (21D5039d), and the installer no longer works. Same error as above. I don't have any antivirus software or special security settings. Let me know if there is anything I could help. Thanks.

dmitshur commented 2 years ago

Thanks for testing and confirming this Cherry. Can you try some other macOS installers (e.g., Chrome) and see if they're also affected, or it it only happening to the Go installer?

Given how generic the error message is, it's hard to tell whether it's a bug in macOS or if its requirements changed. I'll try checking if there are any documented changes happening in 12.2 that might explain this.

cherrymui commented 2 years ago

Chrome download gives a dmg, instead of pkg, so I guess it isn't really comparable. I tried pkgs for Zoom and MacTex, and both work. So it seems not all pkgs stop working. Do you know if there is any software from Google that is in pkg form?

dmitshur commented 2 years ago

I found that grr has a .pkg installer for macOS at https://grr-doc.readthedocs.io/en/latest/deploying-grr-clients/overview.html#downloading-clients, although it seems to download the client you may need to run the server, making it somewhat trickier (the quickstart estimates that it can be done in 5 minutes).

Developer-Ecosystem-Engineering commented 2 years ago

Thanks we are aware of this issue

dmitshur commented 2 years ago

As it was helpfully pointed out, this may be related to our .pkg having a duplicate file in its table of content:

$ xar -t -f ./go1.17.6.darwin-arm64.pkg
[...]
Resources/bg.png
Resources/bg.png
[...]

A much older .pkg, such as go1.11.1.darwin-amd64.pkg, doesn't have that problem. I'll take a look at how we're building the .pkg file to see what might be causing the duplicate file to appear.

dmitshur commented 2 years ago

Bisection shows the duplicate bg.png file was introduced in the change we made to fix issue #44239, in Feb 2021. The .pkg installer for go1.15.8 (released just before that change) has one bg.png, but go1.15.9 has two.

That fix involved one line being added to the distribution XML to specify the "background-darkAqua" property:

 <?xml version="1.0" encoding="utf-8" standalone="no"?>
 <installer-gui-script minSpecVersion="1">
   <title>Go</title>
   <background mime-type="image/png" file="bg.png" alignment="left" />
+  <background-darkAqua mime-type="image/png" file="bg.png" alignment="left" />
   <options hostArchitectures="x86_64" customize="never" allow-external-scripts="no" />
 [...]

It seems that reusing the same bg.png file for both "background" and "background-darkAqua" properties may be causing our invocation of the productbuild command (or possibly another, later command in the signing and notarization sequence) to construct a .pkg with two Resources/bg.png entries.

Based on that, I think if we use two different background image files for light and dark modes, it might be sufficient to avoid this problem.

dmitshur commented 2 years ago

We've applied the aforementioned change to our release pipeline, so macOS installers that are published alongside upcoming Go releases should not be affected by this problem. We can keep this issue open until the next Go release is out, possibly Go 1.18 Beta 2, to verify.

Thanks for reporting this.

dmitshur commented 2 years ago

This issue doesn't reproduce on the final public release of macOS 12.2 (21D49) that was released last week, so the existing Go installers continue to work with it. (Thanks @Developer-Ecosystem-Engineering!)

Go 1.18 Beta 2 came out today, and the change we've applied is having the intended effect, as reported by:

$ xar -t -f ./go1.18beta2.darwin-arm64.pkg | sort | uniq -d | wc -l
       0

# Compare with:
$ xar -t -f ./go1.18beta1.darwin-arm64.pkg | sort | uniq -d | wc -l
       1

If/when we find a way to use productbuild to specify a single background image for both light and dark appearance without it producing a duplicate file, we can revert our change, since it adds 10 KB unnecessarily (but 10 KB is negligible compared to the total installer size). Until then it's fine to keep it.

It seems there's nothing more to do here, so closing. Thanks again to @alaroldai for the well-timed report and @cherrymui for helping.

dmitshur commented 7 months ago

If/when we find a way to use productbuild to specify a single background image for both light and dark appearance without it producing a duplicate file, we can revert our change, since it adds 10 KB unnecessarily (but 10 KB is negligible compared to the total installer size). Until then it's fine to keep it.

A follow up on this: by now the underlying bug is resolved, and we've been able to drop one of the two identical background images during the work for #63147, in CL 550321 and CL 552016.

That change took effect starting with Go 1.22 RC 2 and onwards, and there hasn't been anything unexpected uncovered.

$ xar -t -f go1.22rc1.darwin-arm64.pkg | grep 'Resources/'
Resources/bg-light.png
Resources/bg-dark.png
$ xar -t -f go1.22rc2.darwin-arm64.pkg | grep 'Resources/'
Resources/background.png

# Still 0 duplicate files in both cases.
$ xar -t -f go1.22rc1.darwin-arm64.pkg | sort | uniq -d | wc -l
       0
$ xar -t -f go1.22rc2.darwin-arm64.pkg | sort | uniq -d | wc -l
       0

golang / go