search

golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
124.04k stars 17.67k forks source link

x/mobile: Unable to build xcframework with gomobile and Xcode 14 #53316

Closed darrarski closed 4 weeks ago

darrarski commented 2 years ago

What version of Go are you using (go version)?

$ go version
go version go1.18.3 darwin/arm64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/darrarski/Library/Caches/go-build"
GOENV="/Users/darrarski/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/darrarski/.go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/darrarski/.go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/opt/homebrew/opt/go/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/opt/homebrew/opt/go/libexec/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.18.3"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/darrarski/Dev/repo/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/bh/t75dvhh939lbvnsq6n60mgtr0000gn/T/go-build841782302=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I tried to build xcframework for iOS using Xcode 14 (14A5228q) developer tools with the following command:

$ gomobile bind -target ios gitlab.com/elixxir/client/bindings

What did you expect to see?

I expected the command to create xcframework. It works when I switch to Xcode 13.4.1 (13F100) developer tools.

What did you see instead?

When using Xcode 14 (14A5228q) developer tools, the command fails with the following error:

$ gomobile bind -target ios gitlab.com/elixxir/client/bindings
gomobile: ios/arm64: go build -buildmode=c-archive -o /var/folders/bh/t75dvhh939lbvnsq6n60mgtr0000gn/T/gomobile-work-1445876485/bindings-ios-arm64.a ./gobind failed: exit status 2
# runtime/cgo
cgo: C compiler "2022-06-09" not found: exec: "2022-06-09": executable file not found in $PATH
cherrymui commented 2 years ago

cgo: C compiler "2022-06-09"

I think gomobile picks up the C compiler from running xcrun --find clang, and that command on darwin/arm64 can sometimes contains bogus logging output starting with the date and time. That probably confuses gomobile.

Could you try running that command again, maybe twice?

gomobile should probably be more resilient on that. Maybe it should only use stdout of xcrun, not stderr. cc @hyangah @golang/ios

changkun commented 2 years ago

I plan to upgrade to Xcode 14 soon so that it may be possible for me to take a look.

darrarski commented 2 years ago

I tried to run the command several times yesterday, but the result was always the same. I'm unfamiliar with golang, but the error message "exec: 2022-06-09 not found" seemed suspicious. I thought it could be some kind of parsing issue.

I'm not sure if the problem is caused by parsing the result of xcrun --find clang, which gives the following output:

with Xcode 13.4.1 (13F100):

$ xcrun --find clang
/Applications/Xcode-13.4.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang

with Xcode 14 (14A5228q):

$ xcrun --find clang
/Applications/Xcode-14.0.0-Beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang

However, I rerun the command today, and... it works! Still, I believe there is a bug hiding somewhere, as the problem was probably caused by the given date when the command was run (2022-06-09). Thankfully today (2022-06-10) it works fine :-)

darrarski commented 2 years ago

I plan to upgrade to Xcode 14 soon so that it may be possible for me to take a look.

I'm afraid you'll need to go back in time to reproduce ;-)

cherrymui commented 2 years ago

On my mac M1 machine, it sometimes print messages like

2022-06-07 22:25:25.471 xcodebuild[8405:27734] Requested but did not find extension point with identifier Xcode.IDEKit.ExtensionSentinelHostApplications for extension Xcode.DebuggerFoundation.AppExtensionHosts.watchOS of plug-in com.apple.dt.IDEWatchSupportCore
2022-06-07 22:25:25.471 xcodebuild[8405:27734] Requested but did not find extension point with identifier Xcode.IDEKit.ExtensionPointIdentifierToBundleIdentifier for extension Xcode.DebuggerFoundation.AppExtensionToBundleIdentifierMap.watchOS of plug-in com.apple.dt.IDEWatchSupportCore

It doesn't always print that. In my experience it is more likely to print if it is the first time to use such commands in a while. And the second and subsequent runs just behave normally, without the extra output.

Maybe it will print if you restart the machine and run that command immediately.

jakubgs commented 2 years ago

I have also experienced this issue with the following messages:

 % xcrun --find clang
2022-09-07 14:27:45.757 xcodebuild[6265:9851879] Requested but did not find extension point with identifier Xcode.IDEKit.ExtensionSentinelHostApplications for extension Xcode.DebuggerFoundation.AppExtensionHosts.watchOS of plug-in com.apple.dt.IDEWatchSupportCore
2022-09-07 14:27:45.757 xcodebuild[6265:9851879] Requested but did not find extension point with identifier Xcode.IDEKit.ExtensionPointIdentifierToBundleIdentifier for extension Xcode.DebuggerFoundation.AppExtensionToBundleIdentifierMap.watchOS of plug-in com.apple.dt.IDEWatchSupportCore
2022-09-07 14:27:45.813 xcodebuild[6265:9851879] XType: com.apple.fonts is not accessible.
2022-09-07 14:27:45.813 xcodebuild[6265:9851879] XType: XTFontStaticRegistry is enabled.
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang

These warnings appear to be included in stdout instead of stderr, and hence they affect how the path is parsed.

I have currently no idea how to fix those warnings.

A proper fix would probably involve parsing and matching all lines with a regex.

jakubgs commented 2 years ago

The issue appears to be coming from envClang() function:

func envClang(sdkName string) (clang, cflags string, err error) {
    if buildN {
        return sdkName + "-clang", "-isysroot " + sdkName, nil
    }
    cmd := exec.Command("xcrun", "--sdk", sdkName, "--find", "clang")
    out, err := cmd.CombinedOutput()
    if err != nil {
        return "", "", fmt.Errorf("xcrun --find: %v\n%s", err, out)
    }
    clang = strings.TrimSpace(string(out))

https://github.com/golang/mobile/blob/8578da9835fd365e78a6e63048c103b27a53a82c/cmd/gomobile/env.go#L434-L443

Which does indeed seem to call xcrun --find clang and accepts the output from cmd.CombinedOutput() indiscriminately.

jakubgs commented 2 years ago

Based on my tests this appears to only happen on the first try:

Last login: Wed Sep  7 09:14:57 2022 from 12.34.56.67
 % xcrun --find clang
2022-09-07 14:50:13.907 xcodebuild[69942:386823822] Requested but did not find extension point with identifier Xcode.IDEKit.ExtensionSentinelHostApplications for extension Xcode.DebuggerFoundation.AppExtensionHosts.watchOS of plug-in com.apple.dt.IDEWatchSupportCore
2022-09-07 14:50:13.908 xcodebuild[69942:386823822] Requested but did not find extension point with identifier Xcode.IDEKit.ExtensionPointIdentifierToBundleIdentifier for extension Xcode.DebuggerFoundation.AppExtensionToBundleIdentifierMap.watchOS of plug-in com.apple.dt.IDEWatchSupportCore
2022-09-07 14:50:14.041 xcodebuild[69942:386823822] XType: com.apple.fonts is not accessible.
2022-09-07 14:50:14.041 xcodebuild[69942:386823822] XType: XTFontStaticRegistry is enabled.
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang

 % xcrun --find clang
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang

 % xcrun --find clang
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang

 % xcrun --find clang
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang

After host reboot it resets and those log messages show up again.

jakubgs commented 2 years ago

According to this discussion: https://developer.apple.com/forums/thread/703233 The fix is:

Remove CommandLineTools

sudo rm -rf /Library/Developer/CommandLineTools

Reinstall CommandLineTools

xcode-select --install

Select CommandLineTools

sudo xcode-select -s /Library/Developer/CommandLineTools

But I tried reinstalling Command Line Tools or installing newer than 13.3.1 - in my case 13.4 - and the message is still there.

Honestly some input validation could be added to envClang.

darrarski commented 2 years ago

Small update from my side:

Not sure what changed, but now I'm able to run gomobile bind with the following configuration:

No more "cgo: C compiler" errors occurs when running gomobile bind. It binds successfully, however, the produced xcframework is not usable. When trying to build iOS or macOS application that uses the framework, I get this error:

Undefined symbols for architecture arm64:
  "_objc_msgSend$initWithObject:", referenced from:
      -[RefTracker assignRefnumAndIncRefcount:] in Bindings(000007.o)
  "_objc_msgSend$setObject:forKeyedSubscript:", referenced from:
      -[RefTracker assignRefnumAndIncRefcount:] in Bindings(000007.o)
  "_objc_msgSend$intValue", referenced from:
  ...

I truncated it because it just lists all functions and methods that gomobile exposes from go to Objective-C.

When creating the same xcframework with Xcode 13.4.1, everything works fine, without any issues.

UPDATE: Because the above problem is probably not related to the original one (with gomobile bing failed to run). I created a new issue for it: #55028

UPDATE: There was something messed up with my local environment. I reinstalled golang and can't reproduce the issue anymore. Sorry for the confusion.

I can now use gomobile bind without any issues on the following setup:

Not sure what resolved the original issue I reported here, but I think we can now close it.

jakubgs commented 2 years ago

@darrarski please do not close this, this is a valid issue I'm fixing in:

darrarski commented 2 years ago

@darrarski please do not close this, this is a valid issue I'm fixing in:

Oh, OK, sorry for that. Will keep an eye on the progress, but like I said - it works now for me with latest Xcode and gomobile. Thanks a lot for all your precious time and effort!

johnmaguire commented 1 month ago

We're hitting a variation of this issue as well:

# runtime/cgo
cgo: C compiler "Warning:" not found: exec: "Warning:": executable file not found in $PATH
runtime/cgo
runtime/cgo
# runtime/cgo
cgo: C compiler "Warning:" not found: exec: "Warning:": executable file not found in $PATH
# runtime/cgo
cgo: C compiler "Warning:" not found: exec: "Warning:": executable file not found in $PATH
gomobile: iossimulator/arm64: go build -v -trimpath -buildmode=c-archive -o /var/folders/06/554jgqx57blcwbchmjmkts600000gn/T/gomobile-work-2147008467/mobileNebula-iossimulator-arm64.a ./gobind failed: exit status 1

make: *** [MobileNebula.xcframework] Error 1

We were able to stop the warnings in the meantime with:

unexport SWIFT_DEBUG_INFORMATION_VERSION
unexport SWIFT_DEBUG_INFORMATION_FORMAT

We believe these environment variables are set by Xcode.

The warnings are emitted when running gomobile with the environment variables set, and no flags: 

❯ SWIFT_DEBUG_INFORMATION_VERSION="hi john" gomobile >/dev/null
Warning: unknown environment variable SWIFT_DEBUG_INFORMATION_VERSION
Gomobile is a tool for building and running mobile apps written in Go.

...snip...

This comes from xcrun:

❯ SWIFT_DEBUG_INFORMATION_VERSION="hi john" xcrun --find clang
Warning: unknown environment variable SWIFT_DEBUG_INFORMATION_VERSION
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang

The fix in https://github.com/golang/mobile/pull/84 looks like it would solve our problem.

jakubgs commented 1 month ago

I don' think anyone cares. I've submitted a fix for this 2 years ago and nobody has even reviewed it:

We just apply a patch to this tool at build time through Nix, because this will probably never get merged: https://github.com/status-im/status-mobile/blob/cfd30e0d2401bd61a519eba5d9f467b23120f721/nix/overlay.nix#L68-L71

hajimehoshi commented 4 weeks ago

@jakubgs

These warnings appear to be included in stdout instead of stderr, and hence they affect how the path is parsed.

How did you check this? What was the result of xcrun --find clang > /tmp/out and cat /tmp/out?

johnmaguire commented 4 weeks ago

In my own testing, xcrun appears to print warnings to stderr, yet they interfere with the gomobile build anyway:

❯ SWIFT_DEBUG_INFORMATION_VERSION="hi john" xcrun --find clang >/dev/null
Warning: unknown environment variable SWIFT_DEBUG_INFORMATION_VERSION
hajimehoshi commented 4 weeks ago

❯ SWIFT_DEBUG_INFORMATION_VERSION="hi john" xcrun --find clang >/dev/null

I cannot distinguish whether xcrun output that warning or not from the command... What was the result of xcrun --find clang > /tmp/out and cat /tmp/out?

johnmaguire commented 4 weeks ago

@hajimehoshi >/dev/null redirects stdout to /dev/null. That means the only thing printed is stderr.

❯ SWIFT_DEBUG_INFORMATION_VERSION="hi john" xcrun --find clang >/tmp/out
Warning: unknown environment variable SWIFT_DEBUG_INFORMATION_VERSION
❯ cat /tmp/out
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang

Do you understand?

johnmaguire commented 4 weeks ago

I do not have this issue:

gomobile: ios/arm64: go build -buildmode=c-archive -o /var/folders/bh/t75dvhh939lbvnsq6n60mgtr0000gn/T/gomobile-work-1445876485/bindings-ios-arm64.a ./gobind failed: exit status 2
# runtime/cgo
cgo: C compiler "2022-06-09" not found: exec: "2022-06-09": executable file not found in $PATH

I have a related issue, per my original comment https://github.com/golang/go/issues/53316#issuecomment-2407626810 which is also fixed by the linked PR.

hajimehoshi commented 4 weeks ago

@johnmaguire So, as you already said, the warning at https://github.com/golang/go/issues/53316#issuecomment-1239392434 was not output, which means you have not reproduced the case.

2022-09-07 14:27:45.757 xcodebuild[6265:9851879] Requested but did not find extension point with identifier Xcode.IDEKit.ExtensionSentinelHostApplications for extension Xcode.DebuggerFoundation.AppExtensionHosts.watchOS of plug-in com.apple.dt.IDEWatchSupportCore
2022-09-07 14:27:45.757 xcodebuild[6265:9851879] Requested but did not find extension point with identifier Xcode.IDEKit.ExtensionPointIdentifierToBundleIdentifier for extension Xcode.DebuggerFoundation.AppExtensionToBundleIdentifierMap.watchOS of plug-in com.apple.dt.IDEWatchSupportCore
2022-09-07 14:27:45.813 xcodebuild[6265:9851879] XType: com.apple.fonts is not accessible.
2022-09-07 14:27:45.813 xcodebuild[6265:9851879] XType: XTFontStaticRegistry is enabled.

By the way, I cannot reproduce the issue either, so I cannot test this.

hajimehoshi commented 4 weeks ago

https://go-review.googlesource.com/c/mobile/+/432657/comment/571d2b9a_3519aa82/

My intuition is that just replacing CombinedOutput with Output should fix the issue instead of adding a filtering function, so I would like to check whether the warning was really output to stdout, not stderr @jakubgs

johnmaguire commented 4 weeks ago

@hajimehoshi I apologize - I did not read the linked PR (only the title) and I understand the confusion now. I agree that replacing CombinedOutput with Output is a much cleaner fix for my issue.

@jakubgs will need to test the command you suggested, as I cannot reproduce the messages he is seeing.

jakubgs commented 4 weeks ago

I think that depends on Xcode version and other factors.

jakubgs@macm1-01.ih-eu-mda1.ci.devel:~ % SWIFT_DEBUG_INFORMATION_VERSION="hi john" xcrun --find clang >/dev/null 
Warning: unknown environment variable SWIFT_DEBUG_INFORMATION_VERSION
jakubgs@macm1-01.ih-eu-mda1.ci.devel:~ % SWIFT_DEBUG_INFORMATION_VERSION="hi john" xcrun --find clang            
Warning: unknown environment variable SWIFT_DEBUG_INFORMATION_VERSION
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang

You cannot expect Apple to be consistent. My fix is still necessary I think, as Apple is a garbage company.

hajimehoshi commented 4 weeks ago

@jakubgs So would it be possible to reproduce the debug message 2022-09-07 14:27:45.757 xcodebuild[6265:9851879] Requested but did not find extension point with identifier Xcode.IDEKit.ExtensionSentinelHostApplications for extension Xcode.DebuggerFoundation.AppExtensionHosts.watchOS of plug-in com.apple.dt.IDEWatchSupportCore ... output to stdout, not stderr?

jakubgs commented 4 weeks ago

As I said, it's xcode version dependent.

hajimehoshi commented 4 weeks ago

So we can no longer confirm the debug message was really output to stdout, not stderr.

I'm very skeptical about this (the debug message was output to stdout). We should go with replacing CombinedOutput with Output instead of a filtering function in your CL.

jakubgs commented 4 weeks ago

The issue is clear. We had this problem, which is why we applied the patch which has fixed the issue: https://github.com/status-im/status-mobile/blob/cfd30e0d2401bd61a519eba5d9f467b23120f721/nix/overlay.nix#L68-L71

You can see the error we got here:

 % xcrun --find clang
2022-09-07 14:50:13.907 xcodebuild[69942:386823822] Requested but did not find extension point with identifier Xcode.IDEKit.ExtensionSentinelHostApplications for extension Xcode.DebuggerFoundation.AppExtensionHosts.watchOS of plug-in com.apple.dt.IDEWatchSupportCore
2022-09-07 14:50:13.908 xcodebuild[69942:386823822] Requested but did not find extension point with identifier Xcode.IDEKit.ExtensionPointIdentifierToBundleIdentifier for extension Xcode.DebuggerFoundation.AppExtensionToBundleIdentifierMap.watchOS of plug-in com.apple.dt.IDEWatchSupportCore
2022-09-07 14:50:14.041 xcodebuild[69942:386823822] XType: com.apple.fonts is not accessible.
2022-09-07 14:50:14.041 xcodebuild[69942:386823822] XType: XTFontStaticRegistry is enabled.
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang

And the details in the issue and in the PR:

As we can see Apple is garbage and their tools spew nonsense into stdout. This has happened in the past and will happen again in the future. If you refuse to fix this then that's your problem. We use Nix so we can patch this ourselves without problems, but others might not be as lucky.

hajimehoshi commented 4 weeks ago

As we can see Apple is garbage and their tools spew nonsense into stdout.

As I said, I'm very skeptical about this assumption. I don't think the tool output the warning/debug messages to stdout.

jakubgs commented 4 weeks ago

It does. We fixed it with the patch I've linked. You can't deny it, you can just call me a liar. Have fun maintaining software this way.

hajimehoshi commented 4 weeks ago

It does.

But we cannot reproduce that, right?

jakubgs commented 4 weeks ago

You can, just get an older Xcode version. And probably a matching OSX.

hajimehoshi commented 4 weeks ago

You can't deny it, you can just call me a liar. Have fun maintaining software this way.

I'm just pointing out that you might be misunderstanding. From the comments https://github.com/golang/go/issues/53316#issuecomment-1239392434 and https://github.com/golang/go/issues/53316#issuecomment-1239398184, the stdout and stderr seemed to be mixed so we cannot say the debug message was really output to stdout.

I'm fine to help you if we go with replacing CombinedOutput with Output. If you don't want, I'll make my CL to close this issue.

jakubgs commented 4 weeks ago

I have no interest nor time to reproduce this for you. As I said, we patched it thanks to Nix being amazing.

The issue is real and will show up again, whether with older or newer combinations of Xcode and OSX. I would not close it if I were you, but you do you.

hajimehoshi commented 4 weeks ago

Sure, I'll do :-)

gopherbot commented 4 weeks ago

Change https://go.dev/cl/620315 mentions this issue: cmd/gomobile: use Output instead of CombinedOutput at envClang