cmd/go: support passing @args-file to -toolexec commands

RomainMuller commented 2 hours ago

Go version

go version go1.23.2 linux/arm64

Output of `go env` in your module/workspace:

GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_arm64'
GOVCS=''
GOVERSION='go1.23.2'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/root/.config/go/telemetry'
GCCGO='gccgo'
GOARM64='v8.0'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/Users/romain.marcadier/Development/Datadog/orchestrion/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build89119916=/tmp/go-build -gno-record-gcc-switches'

What did you do?

The following command works as intended:

$ go build -a github.com/elastic/go-elasticsearch/v8/typedapi/types

However if you use any -toolexec value, it suddenly fails on Windows:

$ go build -toolexec="D:\\a\\orchestrion\\orchestrion\\bin\\orchestrion.exe" -a github.com/elastic/go-elasticsearch/v8/typedapi/types
fork/exec D:\\a\\orchestrion\\orchestrion\\bin\\orchestrion.exe: The filename or extension is too long.

This is because the github.com/elastic/go-elasticsearch/v8/typedapi/types package contains a metric ton of files, and results in an excessively long command line for Windows.

What did you see happen?

What we observe is that when there is no -toolexec the Windows arguments size limit (of 32KiB) is "worked around" by passing an @args-file value that contains all the arguments instead of passing the arguments list itself directly...

Unfortunately, the use of -toolexec appears to disable this mechanism, making it impossible to build packages with very large amounts of files in them on Windows.

I have verified this behavior using strace on a Linux machine... I expect the same behavior happens on Windows as well and explains our issue here...

What did you expect to see?

I expect a package that successfully builds without -toolexec to also be build-able with -toolexec, and for the -toolexec tool to actually also receive the arguments file stand-in when appropriate.

gabyhelp commented 2 hours ago

Related Issues and Documentation

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

RomainMuller commented 2 hours ago

As suggested by @gabyhelp, this is actually very much related to (although this has the added characteristic of being an apparent -toolexec break out of the response file feature):

RomainMuller commented 2 hours ago

Also, the culprit is likely this code path:

https://github.com/golang/go/blob/e5e552b816793f2b5729744c27995cce12baf52e/src/cmd/go/internal/work/exec.go#L3431-L3435

Since in our case, the prog ends up being the -toolexec executable...

golang / go