search

golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
123.64k stars 17.61k forks source link

cmd/go: ethereum build not reproducible #59500

Closed jrick closed 1 year ago

jrick commented 1 year ago

What version of Go are you using (go version)?

$ go version
go version go1.20.3 windows/amd64

$ go version
go version go1.20.3 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env (windows) Output
$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\jrick\AppData\Local\go-build
set GOENV=C:\Users\jrick\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\jrick\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\jrick\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Program Files\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.20.3
set GCCGO=gccgo
set GOAMD64=v1
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=0
set GOMOD=C:\Users\jrick\src\go-ethereum\go.mod
set GOWORK=
set CGO_CFLAGS=-O2 -g
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-O2 -g
set CGO_FFLAGS=-O2 -g
set CGO_LDFLAGS=-O2 -g
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=C:\Users\jrick\AppData\Local\Temp\go-build1436174171=/tmp/go-build -gno-record-gcc-switches
go env (linux) Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/jrick/.cache/go-build"
GOENV="/home/jrick/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/jrick/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/jrick/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib64/go/1.20"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib64/go/1.20/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.3"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/jrick/src/go-ethereum/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build632484733=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Performing a cross compile, that should be bit-for-bit reproducible, of the geth executable from the go-ethereum repo checked out at commit b946b7a13b749c99979e312c83dce34cac8dd7b1 results in different file hashes when building on Linux vs Windows.

$ go version && GOOS=windows GOARCH=amd64 CGO_ENABLED=0 GOFLAGS= GOWORK=off go build -buildvcs=false  -trimpath -ldflags='-buildid= -s -w' -o geth.exe ./cmd/geth && ls -l geth.exe && openssl sha256 geth.exe
go version go1.20.3 windows/amd64
-rwxr-xr-x 1 jrick 197609 40166400 Apr  8 08:32 ./geth.exe*
SHA256(geth.exe)= 6d065b5978219c934d688e952fd658a58538d324c0e206f0c46f253e43b5a39a

$ go version && GOOS=windows GOARCH=amd64 CGO_ENABLED=0 GOFLAGS= GOWORK=off go build -buildvcs=false  -trimpath -ldflags='-buildid= -s -w' -o geth.exe ./cmd/geth && ls -l geth.exe && openssl sha256 geth.exe
go version go1.20.3 linux/amd64
-rwxr-xr-x 1 jrick jrick 40152064 Apr  8 12:31 geth.exe
SHA2-256(geth.exe)= 6c1e0ff5cb2c8c215a26a47da2a171312b4b5aaa588672d1125bb55548e1aa75

What did you expect to see?

Same executable built form both hosts. These are the same procedures we use for reproducible cross compiles of other packages, and have not had an issue until building this project.

What did you see instead?

Differing builds

jrick commented 1 year ago

Some additional info: I can reproduce the same build as done by Linux from OpenBSD. The Windows build is the odd one out.

$ go version && GOOS=windows GOARCH=amd64 CGO_ENABLED=0 GOFLAGS= GOWORK=off go build -buildvcs=false  -trimpath -ldflags='-buildid= -s -w' -o geth.exe ./cmd/geth && ls -l geth.exe && openssl sha256 geth.exe 
go version go1.20.3 openbsd/amd64
-rwxr-xr-x  1 jrick  jrick  40152064 Apr  8 08:55 geth.exe*
SHA256(geth.exe)= 6c1e0ff5cb2c8c215a26a47da2a171312b4b5aaa588672d1125bb55548e1aa75
seankhliao commented 1 year ago

can you get the go version -m $binary output of the built artifacts?

jrick commented 1 year ago

Sure. The output is the same for both.

./geth.exe: go1.20.3
        path    github.com/ethereum/go-ethereum/cmd/geth
        mod     github.com/ethereum/go-ethereum (devel)
        dep     github.com/StackExchange/wmi    v0.0.0-20180116203802-5d049714c4a6      h1:fLjPD/aNc3UIOA6tDi6QXUemppXK3P9BI7mr2hd6gx8=
        dep     github.com/VictoriaMetrics/fastcache    v1.6.0  h1:C/3Oi3EiBCqufydp1neRZkqcwmEiuRT9c3fqvvgKm5o=
        dep     github.com/beorn7/perks v1.0.1  h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
        dep     github.com/btcsuite/btcd/btcec/v2       v2.2.0  h1:fzn1qaOt32TuLjFlkzYSsBC35Q3KUjT1SwPxiMSCF5k=
        dep     github.com/cespare/xxhash/v2    v2.2.0  h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44=
        dep     github.com/cockroachdb/errors   v1.9.1  h1:yFVvsI0VxmRShfawbt/laCIDy/mtTqqnvoNgiy5bEV8=
        dep     github.com/cockroachdb/logtags  v0.0.0-20230118201751-21c54148d20b      h1:r6VH0faHjZeQy818SGhaone5OnYfxFR/+AzdY3sf5aE=
        dep     github.com/cockroachdb/pebble   v0.0.0-20230209160836-829675f94811      h1:ytcWPaNPhNoGMWEhDvS3zToKcDpRsLuRolQJBVGdozk=
        dep     github.com/cockroachdb/redact   v1.1.3  h1:AKZds10rFSIj7qADf0g46UixK8NNLwWTNdCIGS5wfSQ=
        dep     github.com/cpuguy83/go-md2man/v2        v2.0.2  h1:p1EgwI/C7NhT0JmVkwCD2ZBK8j4aeHQX2pMHHBfMQ6w=
        dep     github.com/crate-crypto/go-ipa  v0.0.0-20220523130400-f11357ae11c7      h1:6IrxszG5G+O7zhtkWxq6+unVvnrm1fqV2Pe+T95DUzw=
        dep     github.com/davecgh/go-spew      v1.1.1  h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
        dep     github.com/deckarep/golang-set/v2       v2.1.0  h1:g47V4Or+DUdzbs8FxCCmgb6VYd+ptPAngjM6dtGktsI=
        dep     github.com/decred/dcrd/dcrec/secp256k1/v4       v4.0.1  h1:YLtO71vCjJRCBcrPMtQ9nqBsqpA1m5sE92cU+pd5Mcc=
        dep     github.com/deepmap/oapi-codegen v1.8.2  h1:SegyeYGcdi0jLLrpbCMoJxnUUn8GBXHsvr4rbzjuhfU=
        dep     github.com/dlclark/regexp2      v1.7.0  h1:7lJfhqlPssTb1WQx4yvTHN0uElPEv52sbaECrAQxjAo=
        dep     github.com/dop251/goja  v0.0.0-20230122112309-96b1610dd4f7      h1:kgvzE5wLsLa7XKfV85VZl40QXaMCaeFtHpPwJ8fhotY=
        dep     github.com/edsrzf/mmap-go       v1.0.0  h1:CEBF7HpRnUCSJgGUb5h1Gm7e3VkmVDrR8lvWVLtrOFw=
        dep     github.com/fatih/color  v1.7.0  h1:DkWD4oS2D8LGGgTQ6IvwJJXSL5Vp2ffcQg58nFV38Ys=
        dep     github.com/fjl/memsize  v0.0.0-20190710130421-bcb5799ab5e5      h1:FtmdgXiUlNeRsoNMFlKLDt+S+6hbjVMEW6RGQ7aUf7c=
        dep     github.com/gballet/go-libpcsclite       v0.0.0-20190607065134-2772fd86a8ff      h1:tY80oXqGNY4FhTFhk+o9oFHGINQ/+vhlm8HFzi6znCI=
        dep     github.com/gballet/go-verkle    v0.0.0-20220902153445-097bd83b7732      h1:AB7YjNrzlVHsYz06zCULVV2zYCEft82P86dSmtwxKL0=
        dep     github.com/getsentry/sentry-go  v0.18.0 h1:MtBW5H9QgdcJabtZcuJG80BMOwaBpkRDZkxRkNC1sN0=
        dep     github.com/go-ole/go-ole        v1.2.1  h1:2lOsA72HgjxAuMlKpFiCbHTvu44PIVkZ5hqm3RSdI/E=
        dep     github.com/go-sourcemap/sourcemap       v2.1.3+incompatible     h1:W1iEw64niKVGogNgBN3ePyLFfuisuzeidWPMPWmECqU=
        dep     github.com/go-stack/stack       v1.8.1  h1:ntEHSVwIt7PNXNpgPmVfMrNhLtgjlmnZha2kOpuRiDw=
        dep     github.com/gofrs/flock  v0.8.1  h1:+gYjHKf32LDeiEEFhQaotPbLuUXjY5ZqxKgXy7n59aw=
        dep     github.com/gogo/protobuf        v1.3.2  h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
        dep     github.com/golang-jwt/jwt/v4    v4.3.0  h1:kHL1vqdqWNfATmA0FNMdmZNMyZI1U6O31X4rlIPoBog=
        dep     github.com/golang/protobuf      v1.5.2  h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw=
        dep     github.com/golang/snappy        v0.0.5-0.20220116011046-fa5810519dcb    h1:PBC98N2aIaM3XXiurYmW7fx4GZkL8feAMVq7nEjURHk=
        dep     github.com/google/uuid  v1.3.0  h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
        dep     github.com/gorilla/websocket    v1.4.2  h1:+/TMaTYc4QFitKJxsQ7Yye35DkWvkdLcvGKqM+x0Ufc=
        dep     github.com/graph-gophers/graphql-go     v1.3.0  h1:Eb9x/q6MFpCLz7jBCiP/WTxjSDrYLR1QY41SORZyNJ0=
        dep     github.com/hashicorp/go-bexpr   v0.1.10 h1:9kuI5PFotCboP3dkDYFr/wi0gg0QVbSNz5oFRpxn4uE=
        dep     github.com/holiman/bloomfilter/v2       v2.0.3  h1:73e0e/V0tCydx14a0SCYS/EWCxgwLZ18CZcZKVu0fao=
        dep     github.com/holiman/uint256      v1.2.2-0.20230321075855-87b91420868c    h1:DZfsyhDK1hnSS5lH8l+JggqzEleHteTYfutAiVlSUM8=
        dep     github.com/huin/goupnp  v1.0.3  h1:N8No57ls+MnjlB+JPiCVSOyy/ot7MJTqlo7rn+NYSqQ=
        dep     github.com/influxdata/influxdb-client-go/v2     v2.4.0  h1:HGBfZYStlx3Kqvsv1h2pJixbCl/jhnFtxpKFAv9Tu5k=
        dep     github.com/influxdata/influxdb1-client  v0.0.0-20220302092344-a9ab5670611c      h1:qSHzRbhzK8RdXOsAdfDgO49TtqC1oZ+acxPrkfTxcCs=
        dep     github.com/influxdata/line-protocol     v0.0.0-20210311194329-9aa0e372d097      h1:vilfsDSy7TDxedi9gyBkMvAirat/oRcL0lFdJBf6tdM=
        dep     github.com/jackpal/go-nat-pmp   v1.0.2  h1:KzKSgb7qkJvOUTqYl9/Hg/me3pWgBmERKrTGD7BdWus=
        dep     github.com/jedisct1/go-minisign v0.0.0-20190909160543-45766022959e      h1:UvSe12bq+Uj2hWd8aOlwPmoZ+CITRFrdit+sDGfAg8U=
        dep     github.com/karalabe/usb v0.0.2  h1:M6QQBNxF+CQ8OFvxrT90BA0qBOXymndZnk5q235mFc4=
        dep     github.com/klauspost/compress   v1.15.15        h1:EF27CXIuDsYJ6mmvtBRlEuB2UVOqHG1tAXgZ7yIO+lw=
        dep     github.com/kr/pretty    v0.3.1  h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
        dep     github.com/kr/text      v0.2.0  h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
        dep     github.com/mattn/go-colorable   v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA=
        dep     github.com/mattn/go-isatty      v0.0.16 h1:bq3VjFmv/sOjHtdEhmkEV4x1AJtvUvOJ2PFAZ5+peKQ=
        dep     github.com/mattn/go-runewidth   v0.0.9  h1:Lm995f3rfxdpd6TSmuVCHVb/QhupuXlYr8sCI/QdE+0=
        dep     github.com/matttproud/golang_protobuf_extensions        v1.0.4  h1:mmDVorXM7PCGKw94cs5zkfA9PSy5pEvNWRP0ET0TIVo=
        dep     github.com/mitchellh/mapstructure       v1.4.1  h1:CpVNEelQCZBooIPDn+AR3NpivK/TIKU8bDxdASFVQag=
        dep     github.com/mitchellh/pointerstructure   v1.2.0  h1:O+i9nHnXS3l/9Wu7r4NrEdwA2VFTicjUEN1uBnDo34A=
        dep     github.com/naoina/go-stringutil v0.1.0  h1:rCUeRUHjBjGTSHl0VC00jUPLz8/F9dDzYI70Hzifhks=
        dep     github.com/naoina/toml  v0.1.2-0.20170918210437-9fafd6967416    h1:shk/vn9oCoOTmwcouEdwIeOtOGA/ELRUw/GwvxwfT+0=
        dep     github.com/olekukonko/tablewriter       v0.0.5  h1:P2Ga83D34wi1o9J6Wh1mRuqd4mF/x/lgBS7N7AbDhec=
        dep     github.com/opentracing/opentracing-go   v1.1.0  h1:pWlfV3Bxv7k65HYwkikxat0+s3pV4bsqf19k25Ur8rU=
        dep     github.com/peterh/liner v1.1.1-0.20190123174540-a2c9a5303de7    h1:oYW+YCJ1pachXTQmzR3rNLYGGz4g/UgFcjb28p/viDM=
        dep     github.com/pkg/errors   v0.9.1  h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
        dep     github.com/prometheus/client_golang     v1.14.0 h1:nJdhIvne2eSX/XRAFV9PcvFFRbrjbcTUj0VP62TMhnw=
        dep     github.com/prometheus/client_model      v0.3.0  h1:UBgGFHqYdG/TPFD1B1ogZywDqEkwp3fBMvqdiQ7Xew4=
        dep     github.com/prometheus/common    v0.39.0 h1:oOyhkDq05hPZKItWVBkJ6g6AtGxi+fy7F4JvUV8uhsI=
        dep     github.com/rogpeppe/go-internal v1.9.0  h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8=
        dep     github.com/rs/cors      v1.7.0  h1:+88SsELBHx5r+hZ8TCkggzSstaWNbDvThkVK8H6f9ik=
        dep     github.com/russross/blackfriday/v2      v2.1.0  h1:JIOH55/0cWyOuilr9/qlrm0BSXldqnqwMsf35Ld67mk=
        dep     github.com/shirou/gopsutil      v3.21.4-0.20210419000835-c7a38de76ee5+incompatible      h1:Bn1aCHHRnjv4Bl16T8rcaFjYSrGrIZvpiGO6P3Q4GpU=
        dep     github.com/status-im/keycard-go v0.2.0  h1:QDLFswOQu1r5jsycloeQh3bVU8n/NatHHaZobtDnDzA=
        dep     github.com/syndtr/goleveldb     v1.0.1-0.20210819022825-2ae1ddf74ef7    h1:epCh84lMvA70Z7CTTCmYQn2CKbY8j86K7/FAIr141uY=
        dep     github.com/tyler-smith/go-bip39 v1.1.0  h1:5eUemwrMargf3BSLRRCalXT93Ns6pQJIjYQN2nyfOP8=
        dep     github.com/urfave/cli/v2        v2.17.2-0.20221006022127-8f469abc00aa   h1:5SqCsI/2Qya2bCzK15ozrqo2sZxkh0FHynJZOTVoV6Q=
        dep     github.com/xrash/smetrics       v0.0.0-20201216005158-039620a65673      h1:bAn7/zixMGCfxrRTfdpNzjtPYqr8smhKouy9mxVdGPU=
        dep     golang.org/x/crypto     v0.1.0  h1:MDRAIl0xIo9Io2xV565hzXHw3zVseKrJKodhohM5CjU=
        dep     golang.org/x/exp        v0.0.0-20230206171751-46f607a40771      h1:xP7rWLUr1e1n2xkK5YB4LI0hPEy3LJC6Wk+D4pGlOJg=
        dep     golang.org/x/net        v0.8.0  h1:Zrh2ngAOFYneWTAIAPethzeaQLuHwhuBkuV6ZiRnUaQ=
        dep     golang.org/x/sync       v0.1.0  h1:wsuoTGHzEhffawBOhz5CYhcrV4IdKZbEyZjBMuTp12o=
        dep     golang.org/x/sys        v0.6.0  h1:MVltZSvRTcU2ljQOhs94SXPftV6DCNnZViHeQps87pQ=
        dep     golang.org/x/text       v0.8.0  h1:57P1ETyNKtuIjB4SRd15iJxuhj8Gc416Y78H3qgMh68=
        dep     golang.org/x/time       v0.0.0-20220922220347-f3bd1da661af      h1:Yx9k8YCG3dvF87UAn2tu2HQLf2dt/eR1bXxpLMWeH+Y=
        dep     google.golang.org/protobuf      v1.28.1 h1:d0NfwRgPtno5B1Wa6L2DAG+KivqkdutMf1UhdNx175w=
        dep     gopkg.in/natefinch/lumberjack.v2        v2.0.0  h1:1Lc07Kr7qY4U2YPouBjpCLxpiyxIVoxqXgkXLknAOE8=
        dep     gopkg.in/natefinch/npipe.v2     v2.0.0-20160621034901-c1b8fa8bdcce      h1:+JknDZhAj8YMt7GC73Ei8pv4MzjDUNPHgQWJdtMAaDU=
        dep     gopkg.in/yaml.v2        v2.4.0  h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
        build   -buildmode=exe
        build   -compiler=gc
        build   -trimpath=true
        build   CGO_ENABLED=0
        build   GOARCH=amd64
        build   GOOS=windows
        build   GOAMD64=v1
jrick commented 1 year ago

Here's an attempt at diffing the binaries. (I removed the -s -w linker flags for this)

$ go tool objdump geth.exe.linux | sed -E 's/(0x|\+)[0-9a-f]+/0xXXXXXX/g' | cut -f1,3,7 > a
$ go tool objdump geth.exe.windows | sed -E 's/(0x|\+)[0-9a-f]+/0xXXXXXX/g' | cut -f1,3,7 > b
$ diff -u a b

diff output: https://gist.github.com/jrick/c3b3b2ad54212a6a634d66997d4d47b3

jrick commented 1 year ago

The differences in the text compiled for element_mul_amd64.s seems to be the most likely cause?

bcmills commented 1 year ago

I am able to reproduce the 6d065b5978219c934d688e952fd658a58538d324c0e206f0c46f253e43b5a39a checksum on a Windows host, and the 6c1e0ff5cb2c8c215a26a47da2a171312b4b5aaa588672d1125bb55548e1aa75 checksum on a Linux host.

In order to rule out differences in git behavior between the two platforms, I also tried building geth.exe using go install. I was able to reproduce a difference there too: when built on linux I see

bryan@guthrie:~/tmp$ GOOS=windows GOARCH=amd64 GOAMD64=v1 CGO_ENABLED=0 GOFLAGS= GOWORK=off go1.20.3 install -buildvcs=false -trimpath -ldflags='-buildid= -s -w' github.com/ethereum/go-ethereum/cmd/geth@v1.11.6-0.20230405110925-b946b7a13b74 && openssl sha256 $(go env GOPATH)/bin/windows_amd64/geth.exe
SHA256(/home/bryan/bin/windows_amd64/geth.exe)= 826ab71fc227add7d4403a85843e7e6ad6cddc80bdcfa844ba2af18fd9b3d2f6
C:\tmp>"C:\program files\go\bin\go" install -buildvcs=false -trimpath "-ldflags=-buildid= -s -w" github.com/ethereum/go-ethereum/cmd/geth@v1.11.6-0.20230405110925-b946b7a13b74

C:\tmp>"C:\program files\go\bin\go" env GOPATH
C:\Users\bryan\go
bryan@guthrie:/mnt/c/tmp$ openssl sha256 /mnt/c/Users/bryan/go/bin/geth.exe
SHA256(/mnt/c/Users/bryan/go/bin/geth.exe)= 4dd9da4ce478efdedcf0ba42639b5b05d0385ab96d069b9f59a34946c8289775

go version -m confirms that they were build with identical dependency versions and build settings.

bcmills commented 1 year ago

Curiously, building using a Go compiler in a WSL mount of the Windows clone of the repo produces yet a third checksum. 😵‍💫 

bryan@guthrie:/mnt/c/Users/bryan/src/github.com/ethereum/go-ethereum$ go1.20.3 version && GOOS=windows GOARCH=amd64 GOAMD64=v1 CGO_ENABLED=0 GOFLAGS= GOWORK=off go1.20.3 build -buildvcs=false  -trimpath -ldflags='-buildid= -s -w' -o geth.exe ./cmd/geth && ls -l geth.exe && openssl sha256 geth.exe
go version go1.20.3 linux/amd64
-rwxrwxrwx 1 bryan bryan 40166400 Apr 10 11:05 geth.exe
SHA256(geth.exe)= 7875cb8cc4b26fd241f65f7f2ce59c4d9237fad4bd4c9cea029e43435ee701bb
bcmills commented 1 year ago

Output from go install -a -x -buildvcs=false -trimpath "-ldflags=-buildid= -s -w" github.com/ethereum/go-ethereum/cmd/geth@v1.11.6-0.20230405110925-b946b7a13b74 on both Windows and Linux, targeting the same Windows configuration: go-windows.log go-cross.log

Those builds reproduced the 4dd9… checksum on Windows and the 826a… checksum on Linux respectively.

@golang/compiler: at first blush this looks like a compiler and/or linker reproducibility bug to me. Note that these are for pure-Go builds (with CGO_ENABLED=0).

prattmic commented 1 year ago

I'm not 100% sure I built from Windows properly (getting a gomote to download and build was fiddly), but in my binaries, the first difference is in internal/buildcfg.goarm.

When built from Windows:

TEXT internal/buildcfg.goarm(SB) internal/buildcfg/cfg.go
  cfg.go:71             0x86ea00                493b6610                CMPQ 0x10(R14), SP                      
  cfg.go:71             0x86ea04                0f86c7000000            JBE 0x86ead1                            
  cfg.go:71             0x86ea0a                4883ec30                SUBQ $0x30, SP                          
  cfg.go:71             0x86ea0e                48896c2428              MOVQ BP, 0x28(SP)                       
  cfg.go:71             0x86ea13                488d6c2428              LEAQ 0x28(SP), BP                       
  cfg.go:77             0x86ea18                488d05a56a0801          LEAQ go:string.*+8348(SB), AX           
  cfg.go:50             0x86ea1f                bb05000000              MOVL $0x5, BX                           
  cfg.go:50             0x86ea24                e85707c9ff              CALL os.Getenv(SB)                      
  cfg.go:50             0x86ea29                4885db                  TESTQ BX, BX                            
  cfg.go:50             0x86ea2c                7512                    JNE 0x86ea40                            
  cfg.go:50             0x86ea2e                488d0573f34a01          LEAQ runtime.gcbits.*+2488(SB), AX      
  cfg.go:50             0x86ea35                bb01000000              MOVL $0x1, BX                           
  cfg.go:50             0x86ea3a                660f1f440000            NOPW 0(AX)(AX*1)                        
  cfg.go:78             0x86ea40                4883fb01                CMPQ $0x1, BX                           
  cfg.go:78             0x86ea44                753c                    JNE 0x86ea82                            
  cfg.go:78             0x86ea46                803835                  CMPB $0x35, 0(AX)                       
  cfg.go:78             0x86ea49                7428                    JE 0x86ea73                             
  cfg.go:80             0x86ea4b                803836                  CMPB $0x36, 0(AX)                       
  cfg.go:80             0x86ea4e                7414                    JE 0x86ea64                             
  cfg.go:82             0x86ea50                803837                  CMPB $0x37, 0(AX)                       
  cfg.go:82             0x86ea53                752d                    JNE 0x86ea82                            
  cfg.go:83             0x86ea55                b807000000              MOVL $0x7, AX                           
  cfg.go:83             0x86ea5a                488b6c2428              MOVQ 0x28(SP), BP                       
  cfg.go:83             0x86ea5f                4883c430                ADDQ $0x30, SP                          
  cfg.go:83             0x86ea63                c3                      RET                                     
  cfg.go:81             0x86ea64                b806000000              MOVL $0x6, AX                           
  cfg.go:81             0x86ea69                488b6c2428              MOVQ 0x28(SP), BP                       
  cfg.go:81             0x86ea6e                4883c430                ADDQ $0x30, SP                          
  cfg.go:81             0x86ea72                c3                      RET                                     
  cfg.go:79             0x86ea73                b805000000              MOVL $0x5, AX                           
  cfg.go:79             0x86ea78                488b6c2428              MOVQ 0x28(SP), BP                       
  cfg.go:79             0x86ea7d                4883c430                ADDQ $0x30, SP                          
  cfg.go:79             0x86ea81                c3                      RET                                     
  cfg.go:85             0x86ea82                488d050d960b01          LEAQ go:string.*+216174(SB), AX         
  cfg.go:85             0x86ea89                bb1e000000              MOVL $0x1e, BX                          
  cfg.go:85             0x86ea8e                31c9                    XORL CX, CX                             
  cfg.go:85             0x86ea90                31ff                    XORL DI, DI                             
  cfg.go:85             0x86ea92                4889fe                  MOVQ DI, SI                             
  cfg.go:85             0x86ea95                e80688c9ff              CALL fmt.Errorf(SB)                     
  cfg.go:85             0x86ea9a                4889058f1f1902          MOVQ AX, internal/buildcfg.Error(SB)    
  cfg.go:85             0x86eaa1                833d484d1f0200          CMPL $0x0, runtime.writeBarrier(SB)     
  cfg.go:85             0x86eaa8                7509                    JNE 0x86eab3                            
  cfg.go:85             0x86eaaa                48891d871f1902          MOVQ BX, internal/buildcfg.Error+8(SB)  
  cfg.go:85             0x86eab1                eb0c                    JMP 0x86eabf                            
  cfg.go:85             0x86eab3                488d3d7e1f1902          LEAQ internal/buildcfg.Error+8(SB), DI  
  cfg.go:85             0x86eaba                e8a129c0ff              CALL runtime.gcWriteBarrierBX(SB)       
  cfg.go:86             0x86eabf                b907000000              MOVL $0x7, CX                           
  cfg.go:86             0x86eac4                0fb6c1                  MOVZX CL, AX                            
  cfg.go:86             0x86eac7                488b6c2428              MOVQ 0x28(SP), BP                       
  cfg.go:86             0x86eacc                4883c430                ADDQ $0x30, SP                          
  cfg.go:86             0x86ead0                c3                      RET                                     
  cfg.go:71             0x86ead1                e88a06c0ff              CALL runtime.morestack_noctxt.abi0(SB)  
  cfg.go:71             0x86ead6                e925ffffff              JMP internal/buildcfg.goarm(SB)         
  :-1                   0x86eadb                cc                      INT $0x3                                
  :-1                   0x86eadc                cc                      INT $0x3                                
  :-1                   0x86eadd                cc                      INT $0x3                                
  :-1                   0x86eade                cc                      INT $0x3                                
  :-1                   0x86eadf                cc                      INT $0x3

When built from Linux:

TEXT internal/buildcfg.goarm(SB) internal/buildcfg/cfg.go
  cfg.go:71             0x86ea00                493b6610                CMPQ 0x10(R14), SP                              
  cfg.go:71             0x86ea04                0f8621010000            JBE 0x86eb2b                                    
  cfg.go:71             0x86ea0a                4883ec38                SUBQ $0x38, SP                                  
  cfg.go:71             0x86ea0e                48896c2430              MOVQ BP, 0x30(SP)                               
  cfg.go:71             0x86ea13                488d6c2430              LEAQ 0x30(SP), BP                               
  cfg.go:73             0x86ea18                488b0d61201902          MOVQ internal/buildcfg.GOOS(SB), CX             
  cfg.go:73             0x86ea1f                48833d6120190207        CMPQ $0x7, internal/buildcfg.GOOS+8(SB)         
  cfg.go:73             0x86ea27                753d                    JNE 0x86ea66                                    
  cfg.go:73             0x86ea29                8139616e6472            CMPL $0x72646e61, 0(CX)                         
  cfg.go:73             0x86ea2f                7535                    JNE 0x86ea66                                    
  cfg.go:73             0x86ea31                668179046f69            CMPW $0x696f, 0x4(CX)                           
  cfg.go:73             0x86ea37                752d                    JNE 0x86ea66                                    
  cfg.go:73             0x86ea39                80790664                CMPB $0x64, 0x6(CX)                             
  cfg.go:73             0x86ea3d                7527                    JNE 0x86ea66                                    
  cfg.go:73             0x86ea3f                488b0d0a201902          MOVQ internal/buildcfg.GOARCH(SB), CX           
  cfg.go:73             0x86ea46                48833d0a20190203        CMPQ $0x3, internal/buildcfg.GOARCH+8(SB)       
  cfg.go:73             0x86ea4e                7516                    JNE 0x86ea66                                    
  cfg.go:73             0x86ea50                6681396172              CMPW $0x7261, 0(CX)                             
  cfg.go:73             0x86ea55                750f                    JNE 0x86ea66                                    
  cfg.go:73             0x86ea57                8079026d                CMPB $0x6d, 0x2(CX)                             
  cfg.go:73             0x86ea5b                7509                    JNE 0x86ea66                                    
  cfg.go:73             0x86ea5d                488d0544f34a01          LEAQ runtime.gcbits.*+2488(SB), AX              
  cfg.go:73             0x86ea64                eb07                    JMP 0x86ea6d                                    
  cfg.go:73             0x86ea66                488d05e3f24a01          LEAQ runtime.gcbits.*+2400(SB), AX              
  cfg.go:77             0x86ea6d                4889442428              MOVQ AX, 0x28(SP)                               
  cfg.go:50             0x86ea72                bb05000000              MOVL $0x5, BX                                   
  cfg.go:50             0x86ea77                488d05466a0801          LEAQ go:string.*+8348(SB), AX                   
  cfg.go:50             0x86ea7e                6690                    NOPW                                            
  cfg.go:50             0x86ea80                e8fb06c9ff              CALL os.Getenv(SB)                              
  cfg.go:50             0x86ea85                4885db                  TESTQ BX, BX                                    
  cfg.go:50             0x86ea88                750a                    JNE 0x86ea94                                    
  cfg.go:77             0x86ea8a                488b442428              MOVQ 0x28(SP), AX                               
  cfg.go:77             0x86ea8f                bb01000000              MOVL $0x1, BX                                   
  cfg.go:78             0x86ea94                4883fb01                CMPQ $0x1, BX                                   
  cfg.go:78             0x86ea98                753c                    JNE 0x86ead6                                    
  cfg.go:78             0x86ea9a                803835                  CMPB $0x35, 0(AX)                               
  cfg.go:78             0x86ea9d                7428                    JE 0x86eac7                                     
  cfg.go:80             0x86ea9f                803836                  CMPB $0x36, 0(AX)                               
  cfg.go:80             0x86eaa2                7414                    JE 0x86eab8                                     
  cfg.go:82             0x86eaa4                803837                  CMPB $0x37, 0(AX)                               
  cfg.go:82             0x86eaa7                752d                    JNE 0x86ead6                                    
  cfg.go:83             0x86eaa9                b807000000              MOVL $0x7, AX                                   
  cfg.go:83             0x86eaae                488b6c2430              MOVQ 0x30(SP), BP                               
  cfg.go:83             0x86eab3                4883c438                ADDQ $0x38, SP                                  
  cfg.go:83             0x86eab7                c3                      RET                                             
  cfg.go:81             0x86eab8                b806000000              MOVL $0x6, AX                                   
  cfg.go:81             0x86eabd                488b6c2430              MOVQ 0x30(SP), BP                               
  cfg.go:81             0x86eac2                4883c438                ADDQ $0x38, SP                                  
  cfg.go:81             0x86eac6                c3                      RET                                             
  cfg.go:79             0x86eac7                b805000000              MOVL $0x5, AX                                   
  cfg.go:79             0x86eacc                488b6c2430              MOVQ 0x30(SP), BP                               
  cfg.go:79             0x86ead1                4883c438                ADDQ $0x38, SP                                  
  cfg.go:79             0x86ead5                c3                      RET                                             
  cfg.go:85             0x86ead6                488d05b9950b01          LEAQ go:string.*+216174(SB), AX                 
  cfg.go:85             0x86eadd                bb1e000000              MOVL $0x1e, BX                                  
  cfg.go:85             0x86eae2                31c9                    XORL CX, CX                                     
  cfg.go:85             0x86eae4                31ff                    XORL DI, DI                                     
  cfg.go:85             0x86eae6                4889fe                  MOVQ DI, SI                                     
  cfg.go:85             0x86eae9                e8b287c9ff              CALL fmt.Errorf(SB)                             
  cfg.go:85             0x86eaee                4889053b1f1902          MOVQ AX, internal/buildcfg.Error(SB)            
  cfg.go:85             0x86eaf5                833df44c1f0200          CMPL $0x0, runtime.writeBarrier(SB)             
  cfg.go:85             0x86eafc                7509                    JNE 0x86eb07                                    
  cfg.go:85             0x86eafe                48891d331f1902          MOVQ BX, internal/buildcfg.Error+8(SB)          
  cfg.go:85             0x86eb05                eb0c                    JMP 0x86eb13                                    
  cfg.go:85             0x86eb07                488d3d2a1f1902          LEAQ internal/buildcfg.Error+8(SB), DI          
  cfg.go:85             0x86eb0e                e84d29c0ff              CALL runtime.gcWriteBarrierBX(SB)               
  cfg.go:86             0x86eb13                488b4c2428              MOVQ 0x28(SP), CX                               
  cfg.go:86             0x86eb18                0fb609                  MOVZX 0(CX), CX     
  cfg.go:86             0x86eb1b                83c1d0                  ADDL $-0x30, CX                                 
  cfg.go:86             0x86eb1e                0fb6c1                  MOVZX CL, AX                                    
  cfg.go:86             0x86eb21                488b6c2430              MOVQ 0x30(SP), BP                               
  cfg.go:86             0x86eb26                4883c438                ADDQ $0x38, SP                                  
  cfg.go:86             0x86eb2a                c3                      RET                                             
  cfg.go:71             0x86eb2b                e83006c0ff              CALL runtime.morestack_noctxt.abi0(SB)          
  cfg.go:71             0x86eb30                e9cbfeffff              JMP internal/buildcfg.goarm(SB)                 
  :-1                   0x86eb35                cc                      INT $0x3                                        
  :-1                   0x86eb36                cc                      INT $0x3                                        
  :-1                   0x86eb37                cc                      INT $0x3                                        
  :-1                   0x86eb38                cc                      INT $0x3                                        
  :-1                   0x86eb39                cc                      INT $0x3                                        
  :-1                   0x86eb3a                cc                      INT $0x3                                        
  :-1                   0x86eb3b                cc                      INT $0x3                                        
  :-1                   0x86eb3c                cc                      INT $0x3                                        
  :-1                   0x86eb3d                cc                      INT $0x3                                        
  :-1                   0x86eb3e                cc                      INT $0x3                                        
  :-1                   0x86eb3f                cc                      INT $0x3

It looks like on Windows build the GOOS and GOARCH checks were deadcode-eliminated, but not on the Linux build?

prattmic commented 1 year ago

(That deadcode elimination doesn't make sense to me. GOOS and GOARCH are clearly not compile-time constants, as they are dependent on an environment variable)

thanm commented 1 year ago

I poked at this a little as well, but I think the difference with the gomote appears to be due to differences in zbootstrap.go:

$ diff  from-gomote/zbootstrap.go /x/go1.20/src/internal/buildcfg/zbootstrap.go 
9c9
< const defaultGOARM = `7`
---
> const defaultGOARM = `5`
thanm commented 1 year ago

I poked at this a little as well, but I think the difference with the gomote appears to be due to differences in zbootstrap.go:

$ diff  from-gomote/zbootstrap.go /x/go1.20/src/internal/buildcfg/zbootstrap.go 
9c9
< const defaultGOARM = `7`
---
> const defaultGOARM = `5`
bcmills commented 1 year ago

I can reproduce the lack of cross-platform reproducibility for commands in GOROOT too:

bryan@guthrie:~/go$ ./bin/go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/bryan/.cache/go-build"
GOENV="/home/bryan/.config/go/env"
GOEXE=".exe"
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/bryan/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="windows"
GOPATH="/home/bryan"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/bryan/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/bryan/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="devel go1.21-63a08e61bd Mon Apr 10 17:13:41 2023 +0000"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="c++"
CGO_ENABLED="0"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-m64 -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1213732133=/tmp/go-build -gno-record-gcc-switches"

bryan@guthrie:~/go$ ./bin/go install cmd/test2json

bryan@guthrie:~/go$ ./bin/go version -m ./pkg/tool/windows_amd64/test2json.exe
./pkg/tool/windows_amd64/test2json.exe: devel go1.21-63a08e61bd Mon Apr 10 17:13:41 2023 +0000
        path    cmd/test2json
        build   -buildmode=exe
        build   -compiler=gc
        build   CGO_ENABLED=0
        build   GOARCH=amd64
        build   GOOS=windows
        build   GOAMD64=v1

bryan@guthrie:~/go$ md5sum ./pkg/tool/windows_amd64/test2json.exe
d76d967100013d62c763873975b31633  ./pkg/tool/windows_amd64/test2json.exe
bryan@guthrie:/mnt/c/Users/bryan/src/go$ ./bin/go.exe env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\bryan\AppData\Local\go-build
set GOENV=C:\Users\bryan\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\bryan\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\bryan\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Users\bryan\src\go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Users\bryan\src\go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=devel go1.21-63a08e61bd Mon Apr 10 17:13:41 2023 +0000
set GCCGO=gccgo
set GOAMD64=v1
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=0
set GOMOD=NUL
set GOWORK=
set CGO_CFLAGS=-O2 -g
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-O2 -g
set CGO_FFLAGS=-O2 -g
set CGO_LDFLAGS=-O2 -g
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=
0 -ffile-prefix-map=C:\Users\bryan\AppData\Local\Temp\go-build4155707933=/tmp/go-build -gno-record-g
cc-switches

bryan@guthrie:/mnt/c/Users/bryan/src/go$ ./bin/go.exe install cmd/test2json

bryan@guthrie:/mnt/c/Users/bryan/src/go$ ./bin/go.exe version -m ./pkg/tool/windows_amd64/test2json.exe
./pkg/tool/windows_amd64/test2json.exe: devel go1.21-63a08e61bd Mon Apr 10 17:13:41 2023 +0000
        path    cmd/test2json
        build   -buildmode=exe
        build   -compiler=gc
        build   CGO_ENABLED=0
        build   GOARCH=amd64
        build   GOOS=windows
        build   GOAMD64=v1

bryan@guthrie:/mnt/c/Users/bryan/src/go$ md5sum ./pkg/tool/windows_amd64/test2json.exe
67a94ca09975b71050088a0e5dfeeabb  ./pkg/tool/windows_amd64/test2json.exe
thanm commented 1 year ago

Maybe this is the culprit here: https://go.googlesource.com/go/+/5e93a2c2042484ee7a941e967294a5248ab6a593/src/cmd/dist/util.go#379 ?

bcmills commented 1 year ago

@thanm, none of these examples are on arm, and at any rate the value of the default GOARM value shouldn't affect the cmd/test2json binary (nor should they affect the compiled github.com/ethereum/go-ethereum/cmd/geth binary AFAICT).

jrick commented 1 year ago

I can reproduce the lack of cross-platform reproducibility for commands in GOROOT too:

I believe this is because -buildid= isn't used here? We also saw issues with reproducibility when attempting to remove this (my understanding is that it should still be reproducible with it unset). Except for geth (and other programs importing the eth libraries), the builds are reproducible with the buildid set to the zero string.

bcmills commented 1 year ago

@jrick, users should not need to set any -ldflags explicitly in order to get a reproducible build. But I did forget to set -trimpath when comparing cmd/test2json. 😅

bcmills commented 1 year ago

...but the difference does hold with -trimpath set:

bryan@guthrie:~/go$ ./bin/go install -trimpath cmd/test2json

bryan@guthrie:~/go$ ./bin/go version -m ./pkg/tool/windows_amd64/test2json.exe
./pkg/tool/windows_amd64/test2json.exe: devel go1.21-63a08e61bd Mon Apr 10 17:13:41 2023 +0000
        path    cmd/test2json
        build   -buildmode=exe
        build   -compiler=gc
        build   -trimpath=true
        build   CGO_ENABLED=0
        build   GOARCH=amd64
        build   GOOS=windows
        build   GOAMD64=v1

bryan@guthrie:~/go$ md5sum ./pkg/tool/windows_amd64/test2json.exe
947b411aa1eb61f8ad347df0be53e32b  ./pkg/tool/windows_amd64/test2json.exe
bryan@guthrie:/mnt/c/Users/bryan/src/go$ ./bin/go.exe install -trimpath cmd/test2json

bryan@guthrie:/mnt/c/Users/bryan/src/go$ ./bin/go.exe version -m ./pkg/tool/windows_amd64/test2json.exe
./pkg/tool/windows_amd64/test2json.exe: devel go1.21-63a08e61bd Mon Apr 10 17:13:41 2023 +0000
        path    cmd/test2json
        build   -buildmode=exe
        build   -compiler=gc
        build   -trimpath=true
        build   CGO_ENABLED=0
        build   GOARCH=amd64
        build   GOOS=windows
        build   GOAMD64=v1

bryan@guthrie:/mnt/c/Users/bryan/src/go$ md5sum ./pkg/tool/windows_amd64/test2json.exe
ef4b5563e65bf0bdfea675967156f55d  ./pkg/tool/windows_amd64/test2json.exe
thanm commented 1 year ago

@thanm, none of these examples are on arm, and at any rate the value of the default GOARM value shouldn't affect the cmd/test2json binary (nor should they affect the compiled github.com/ethereum/go-ethereum/cmd/geth binary AFAICT).

Executable does contain a copy of the "goarm" function, even if it is not used/relevant. Linking without "-s -w":

 $ objdump -t geth.exe | fgrep goarm
[11898](sec  1)(fl 0x00)(ty  308)(scl   2) (nx 0) 0x000000000046da80 internal/buildcfg.goarm
 $

When zbootstrap.go contains "defaultGoArm = 7" (which is what you get when running make.bat on windows) then the deadcoding that Michael described is legal.

bcmills commented 1 year ago

I still don't think that explains it? I don't see a difference in zbootstrap.go between my Windows and Linux GOROOTs, and that wouldn't affect cmd/test2json anyway.

bryan@guthrie:~/go$ diff -r ./src /mnt/c/Users/bryan/src/go/src
diff '--color=auto' -r ./src/cmd/cgo/zdefaultcc.go /mnt/c/Users/bryan/src/go/src/cmd/cgo/zdefaultcc.go
18c18,22
<       return "c++"
---
>       switch goos {
>       case "darwin", "ios", "freebsd", "openbsd":
>               return "clang++"
>       }
>       return "g++"
diff '--color=auto' -r ./src/cmd/go/internal/cfg/zdefaultcc.go /mnt/c/Users/bryan/src/go/src/cmd/go/internal/cfg/zdefaultcc.go
18c18,22
<       return "c++"
---
>       switch goos {
>       case "darwin", "ios", "freebsd", "openbsd":
>               return "clang++"
>       }
>       return "g++"
Only in /mnt/c/Users/bryan/src/go/src/cmd: .vscode
diff '--color=auto' -r ./src/go/build/zcgo.go /mnt/c/Users/bryan/src/go/src/go/build/zcgo.go
5c5
< const defaultCGO_ENABLED = ""
---
> const defaultCGO_ENABLED = "0"
thanm commented 1 year ago

I agree that with test2json , differences in goarm can't be the culprit.

thanm commented 1 year ago

I don't see a difference in zbootstrap.go between my Windows and Linux GOROOTs

Strange. Perhaps this is some sort of artifact of how gomotes work? What is your "defaultGoArm" setting, just out of curiosity?

bcmills commented 1 year ago

Strange. Perhaps this is some sort of artifact of how gomotes work? What is your "defaultGoArm" setting, just out of curiosity?

My zbootstrap.go has

const defaultGOARM = `7`

But IIUC the defaultGOARM is only supposed to vary when compiling ARM on ARM, so it should also be 7 on your machine unless you're somehow using an arm32 as your development host. 😅

prattmic commented 1 year ago

Under what circumstances do we rewrite zbootstrap.go? I also have defaultGOARM = "5" in zbootstrap.go in my main Go checkout, which has been around for years. In another, newer, checkout, I have defaultGOARM = "7".

Edit: Ignore above, that is because this behavior has changed since 1.20 in https://go.dev/cl/470695.

thanm commented 1 year ago

Thanks for that (CL 470695). Yeah, building on linux from 1.20 release branch I get "defaultGOARM" of 5.

davecgh commented 1 year ago

@jrick, users should not need to set any -ldflags explicitly in order to get a reproducible build. But I did forget to set -trimpath when comparing cmd/test2json. 😅

Here is an easy reproduction case to show that builds are reproducible with -buildid= while they aren't without it (NOTE: This is not for geth as otherwise discussed in the issue, but it might otherwise help too):

Without -buildid=

On windows:
$ git clone https://github.com/decred/dcrd && cd dcrd && git checkout release-v1.7.7
$ go version
go version go1.20.3 windows/amd64
$ GOARCH=amd64 GOOS=windows CGO_ENABLED=0 GOFLAGS= GOWORK=off go build -o ./dcrd.exe -trimpath -tags="safe" -ldflags="-s -w" && openssl sha256 ./dcrd.exe
SHA2-256(./dcrd.exe)= fb827bf1b7413728b9abbfd19f0dfdb0406189f303ddab26eaadf8e269599609

On linux:
$ git clone https://github.com/decred/dcrd && cd dcrd && git checkout release-v1.7.7
$ go version
go version go1.20.3 linux/amd64
$ GOARCH=amd64 GOOS=windows CGO_ENABLED=0 GOFLAGS= GOWORK=off go build -o ./dcrd.exe -trimpath -tags="safe" -ldflags="-s -w" && openssl sha256 ./dcrd.exe
SHA2-256(./dcrd.exe)= c4ed28d82b77078924130130d31b0b60953357e79a4c8a441e4b31ca539cb89d

Now, with -buildid=:

On windows:
$ git clone https://github.com/decred/dcrd && cd dcrd && git checkout release-v1.7.7
$ go version
go version go1.20.3 windows/amd64
$ GOARCH=amd64 GOOS=windows CGO_ENABLED=0 GOFLAGS= GOWORK=off go build -o ./dcrd.exe -trimpath -tags="safe" -ldflags="-buildid= -s -w" && openssl sha256 ./dcrd.exe
SHA2-256(./dcrd.exe)= a71a08b3cf5865d6151379f2b07c966ec821946942b9202940e021106c4cde0e

On linux:
$ git clone https://github.com/decred/dcrd && cd dcrd && git checkout release-v1.7.7
$ go version
go version go1.20.3 linux/amd64
$ GOARCH=amd64 GOOS=windows CGO_ENABLED=0 GOFLAGS= GOWORK=off go build -o ./dcrd.exe -trimpath -tags="safe" -ldflags="-buildid= -s -w" && openssl sha256 ./dcrd.exe
SHA2-256(./dcrd.exe)= a71a08b3cf5865d6151379f2b07c966ec821946942b9202940e021106c4cde0e
bcmills commented 1 year ago

Indeed, and I can confirm that in the test2json case:

bryan@guthrie:/mnt/c/Users/bryan/src/go$ ./bin/go.exe install -trimpath -ldflags='all=-buildid=' cmd
/test2json

bryan@guthrie:/mnt/c/Users/bryan/src/go$ md5sum ./pkg/tool/windows_amd64/test2json.exe
713e2f9d867e7aa7eecd097534735640  ./pkg/tool/windows_amd64/test2json.exe
bryan@guthrie:~/go$ ./bin/go install -trimpath -ldflags='all=-buildid=' cmd/test2json

bryan@guthrie:~/go$ md5sum ./pkg/tool/windows_amd64/test2json.exe
713e2f9d867e7aa7eecd097534735640  ./pkg/tool/windows_amd64/test2json.exe

I'll file a separate bug for that because your original report here already corrects for it.

bcmills commented 1 year ago

(Filed the -buildid issue as #59525.)

bcmills commented 1 year ago

Ok, so I don't know why -s or -w would be needed either. If I build geth without those flags (only -trimpath and -ldflags=-buildid=), then I can use go tool objdump to dump both binaries, and they have a very small diff:

bryan@guthrie:~/tmp$ diff ~/tmp/geth-linux.objdump ~/tmp/geth-windows.objdump
1204668,1204684c1204668,1204684
<   build.go:337                0x83ce96                7508                            JNE 0x83cea0
<   build.go:337                0x83ce98                31db                            XORL BX, BX
<   build.go:337                0x83ce9a                31c0                            XORL AX, AX
<   build.go:337                0x83ce9c                0f1f4000                        NOPL 0(AX)
<   build.go:343                0x83cea0                4883fb01                        CMPQ $0x1, BX
<   build.go:343                0x83cea4                7524                            JNE 0x83ceca
<   build.go:343                0x83cea6                803830                          CMPB $0x30, 0(AX)
<   build.go:343                0x83cea9                750d                            JNE 0x83ceb8
<   build.go:344                0x83ceab                c68424e800000000                MOVB $0x0, 0xe8(SP)
<   build.go:344                0x83ceb3                e9b2000000                      JMP 0x83cf6a
<   build.go:341                0x83ceb8                803831                          CMPB $0x31, 0(AX)
<   build.go:341                0x83cebb                750d                            JNE 0x83ceca
<   build.go:342                0x83cebd                c68424e800000001                MOVB $0x1, 0xe8(SP)
<   build.go:342                0x83cec5                e9a0000000                      JMP 0x83cf6a
<   build.go:347                0x83ceca                4c8b8c24a0000000                MOVQ 0xa0(SP), R9
<   build.go:347                0x83ced2                4c8b842498000000                MOVQ 0x98(SP), R8
<   build.go:347                0x83ceda                660f1f440000                    NOPW 0(AX)(AX*1)
---
>   build.go:337                0x83ce96                750c                            JNE 0x83cea4
>   build.go:337                0x83ce98                bb01000000                      MOVL $0x1, BX
>   build.go:337                0x83ce9d                488d05bc0a4401                  LEAQ runtime.gcbits.*+2240(SB), AX
>   build.go:343                0x83cea4                4883fb01                        CMPQ $0x1, BX
>   build.go:343                0x83cea8                7525                            JNE 0x83cecf
>   build.go:343                0x83ceaa                803830                          CMPB $0x30, 0(AX)
>   build.go:343                0x83cead                750d                            JNE 0x83cebc
>   build.go:344                0x83ceaf                c68424e800000000                MOVB $0x0, 0xe8(SP)
>   build.go:344                0x83ceb7                e9ae000000                      JMP 0x83cf6a
>   build.go:341                0x83cebc                803831                          CMPB $0x31, 0(AX)
>   build.go:341                0x83cebf                90                              NOPL
>   build.go:341                0x83cec0                750d                            JNE 0x83cecf
>   build.go:342                0x83cec2                c68424e800000001                MOVB $0x1, 0xe8(SP)
>   build.go:342                0x83ceca                e99b000000                      JMP 0x83cf6a
>   build.go:347                0x83cecf                4c8b8c24a0000000                MOVQ 0xa0(SP), R9
>   build.go:347                0x83ced7                4c8b842498000000                MOVQ 0x98(SP), R8
>   build.go:347                0x83cedf                90                              NOPL
bcmills commented 1 year ago

And that is indeed in the go/build.defaultContext function. 🤔

prattmic commented 1 year ago

Given https://go.dev/cl/470695, does this difference still reproduce at tip? It may be broken in 1.20 but fixed at tip.

bcmills commented 1 year ago

Here we go! https://cs.opensource.google/go/go/+/master:src/go/build/build.go;l=336-339;drc=bd8ec78b08ead1fb34ec8dc7bc4bf2ff7a9e8b82:

    env := os.Getenv("CGO_ENABLED")
    if env == "" {
        env = defaultCGO_ENABLED
    }

And defaultCGO_ENABLED is one of those variables injected by cmd/dist. 😞

bcmills commented 1 year ago

In light of that, I'm gonna say this is arguably a problem in cmd/dist and/or go/build rather than a problem in the compiler or linker. @prattmic and @thanm, thanks for investigating!

cherrymui commented 1 year ago

My guess is that as the build ID is a hash of the source code including imported packages, it includes e.g. defaultGOARM, even if that code is not used in the final binary. That explains why removing build ID with a linker flag makes a difference.

bcmills commented 1 year ago

Given https://go.dev/cl/470695, does this difference still reproduce at tip? It may be broken in 1.20 but fixed at tip.

Building at tip, if I leave CGO_ENABLED unset when running make.bat and make.bash (so that the generated zcgo.go files are the same), then setting -trimpath -ldflags=-buildid= is sufficient to reproduce the same windows binary on both windows and linux.

So it does seem that CL 470695 gets us partway there, in that it reduces one source of non-reproducibility in generated files. Unfortunately, that implies that if you run make.bash on a linux/arm system and then try to cross compile from that to Windows, you will still get a different binary if the program happens to import go/build.

I'll talk to @matloob and @rsc and figure out how we want to proceed.

bcmills commented 1 year ago

My guess is that as the build ID is a hash of the source code including imported packages, it includes e.g. defaultGOARM, even if that code is not used in the final binary. That explains why removing build ID with a linker flag makes a difference.

Unfortunately that doesn't seem to be all of it. Leaving out to the -ldflags=-buildid flag, the binary fails to reproduce — and I've confirmed that geth does not transitively depend on internal/cfg (the only importable package for which the source files differ).

cherrymui commented 1 year ago

and I've confirmed that geth does not transitively depend on internal/cfg

I'm not sure I follow... The go/build package imports internal/buildcfg, and as the issue title said, the binary depends on go/build. (It is buildcfg not cfg.)

bcmills commented 1 year ago

Ah, I misread cmd/go/internal/cfg, which is not really importable by third parties either. 😅

With go/build/zcgo.go fixed, the remaining generated files are in cmd/cgo, cmd/go/internal/cfg, runtime/internal/sys, and time/tzdata.

gopherbot commented 1 year ago

Change https://go.dev/cl/483695 mentions this issue: cmd/dist: rework generated cgo-support logic

bcmills commented 1 year ago

@jrick, @davecgh: can you confirm whether CL 483695 fixes this in your build environment?

Unfortunately I can't think of a good way to test this on the Go builders, since each builder only runs with the output of one make.bash. 🤔

jrick commented 1 year ago

Hmm, it doesn't seem quite right yet.

$ ~/src/go/bin/go version && GOOS=windows GOARCH=amd64 CGO_ENABLED=0 GOFLAGS= GOWORK=off ~/src/go/bin/go build -buildvcs=false  -trimpath '-ldflags=-buildid=' -o geth.exe ./cmd/geth && ls -l geth.exe && openssl sha256 geth.exe
go version devel go1.21-767b3fda4c Tue Apr 11 15:54:42 2023 +0000 windows/amd64
-rwxr-xr-x 1 jrick 197609 55612928 Apr 11 16:30 geth.exe*
SHA256(geth.exe)= fab898c2fc6ddfd488e769ad9c94175d809743e4615b6149abcba1a7b81ae60c

$ ~/src/go/bin/go version && GOOS=windows GOARCH=amd64 CGO_ENABLED=0 GOFLAGS= GOWORK=off ~/src/go/bin/go build -buildvcs=false  -trimpath '-ldflags=-buildid=' -o geth.exe ./cmd/geth && ls -l geth.exe && openssl sha256 geth.exe
go version devel go1.21-767b3fda4c Tue Apr 11 15:54:42 2023 +0000 linux/amd64
-rwxr-xr-x 1 jrick jrick 55598592 Apr 11 16:30 geth.exe
SHA2-256(geth.exe)= 37ea1061d5b7fd7e71e0e86958f0f1818005f2aedba024303a36ab5db00f5910
bcmills commented 1 year ago

Hmm... Do you have any other GO* environment variables set when you run make.bash and/or make.bat on those hosts? (Are there differences in src/internal/buildcfg/zbootstrap.go in the two GOROOTs?)

bcmills commented 1 year ago

(I guess we expect defaultGOOS and defaultGOARCH to differ, but those at least should be pruned out at link time. 🤔)

jrick commented 1 year ago

they are identical

// Code generated by go tool dist; DO NOT EDIT.

package buildcfg

import "runtime"

const defaultGO386 = `sse2`
const defaultGOAMD64 = `v1`
const defaultGOARM = `7`
const defaultGOMIPS = `hardfloat`
const defaultGOMIPS64 = `hardfloat`
const defaultGOPPC64 = `power8`
const defaultGOEXPERIMENT = ``
const defaultGO_EXTLINK_ENABLED = ``
const defaultGO_LDSO = ``
const version = `devel go1.21-767b3fda4c Tue Apr 11 15:54:42 2023 +0000`
const defaultGOOS = runtime.GOOS
const defaultGOARCH = runtime.GOARCH
bcmills commented 1 year ago

Interesting. If you run go tool objdump on the two executables and diff the output, where are the differences reported?

jrick commented 1 year ago

Some of the same differences as my previous objdump diff, but the diff is about half as long now. Still have to mangle the addresses to get a readable diff.

https://gist.github.com/jrick/5ab53af40e430b9aa1f9b4d10b629ddf

jrick commented 1 year ago

Ignoring those instructions that contain addresses that my sed expression didn't rip out, all of the differences appear to be in the compiler output for .s files.

prattmic commented 1 year ago

When I was looking the other day, those .s file differences were because they contain instructions that objdump doesn't know how to decode correctly. When function alignment changes slightly (due to slightly different PC-relative addressing e.g.), then the decoding shifts.

A more robust bet is to look at go tool nm -sort=address and find the first function where the addresses don't exactly match. The function just before that should be the changed function.

bcmills commented 1 year ago

@jrick, is that using objdump or go tool objdump? The latter supports Windows executables, but the former (on Linux) generally does not.

jrick commented 1 year ago

it was go tool objdump

I see this diffing the go tool nm =sort=address output:

  147e560 T github.com/ethereum/go-ethereum/common/lru.(*Cache[go.shape.struct { github.com/ethereum/go-ethereum/eth/gasprice.number uint64; github.com/ethereum/go-ethereum/eth/gasprice.percentiles string },go.shape.struct { github.com/ethereum/go-ethereum/eth/gasprice.reward []*math/big.Int; github.com/ethereum/go-ethereum/eth/gasprice.baseFee *math/big.Int; github.com/ethereum/go-ethereum/eth/gasprice.nextBaseFee *math/big.Int; github.com/ethereum/go-ethereum/eth/gasprice.gasUsedRatio float64 }]).Contains.func1
  147e5c0 T github.com/ethereum/go-ethereum/common/lru.(*Cache[go.shape.struct { github.com/ethereum/go-ethereum/eth/gasprice.number uint64; github.com/ethereum/go-ethereum/eth/gasprice.percentiles string },go.shape.struct { github.com/ethereum/go-ethereum/eth/gasprice.reward []*math/big.Int; github.com/ethereum/go-ethereum/eth/gasprice.baseFee *math/big.Int; github.com/ethereum/go-ethereum/eth/gasprice.nextBaseFee *math/big.Int; github.com/ethereum/go-ethereum/eth/gasprice.gasUsedRatio float64 }]).Add.func1
  147e620 T main.init.0.func1
  147e660 T main.init.0.func2
  147e6a0 T main.init
  1484060 T main.importChain.func2
  14840c0 T main.prepare.func1
  1484120 T type:.eq.main.snapshotIterator
  14841c0 T github.com/ethereum/go-ethereum/ethdb.Database.Get-fm
  1484233 T runtime.etext
  1485000 R runtime.rodata
  1485000 R runtime.types
  1485000 R type:*
  18590a8 R go:string.*
- 1abc148 R go:func.*
- 1c7e138 R runtime.gcbits.*
- 1c81e38 R runtime.gcbss
- 1c82b0a R runtime.gcdata
- 1c82b0a R runtime.egcbss
- 1c83948 R $f32.35800000...