[Bug] GitHub-built .hex file doesn't quite match the one built by `qmk userspace-compile`

[adiabatic]

Describe the Bug

The computer I'm doing all this from is an Apple Silicon-based Mac mini running macOS 14.5.

I've set up a Git clone of qmk_firmware. Now that my keyboard layout is in a userspace repository, the only thing I do with the qmk_firmware clone is run git pull on it and maybe qmk clean.

A couple of days ago, on a lark, I enabled the build_binaries workflow and then manually kicked off a build.

Earlier today, I adjusted a couple keys on my keyboard's layout and then flashed them to my keyboard, and then pushed the changes to my main branch, kicking off a second build. You can see the generated artifacts for this second release on the relevant action's run's page. The download-firmware link is right there on the page. Downloading and extracting this .zip file gave me a file called ergodox_ez_base_zweihander_macos 2.hex (2 because I already have a file with that name downloaded earlier).

Here's what I did just now:

1. in qmk_firmware, run git pull.
2. in qmk_firmware, run qmk clean.
3. in zweihander, run rm ergodox_ez_base_zweihander_macos.hex.
4. in zweihander, run qmk userspace-compile, (re)generating ergodox_ez_base_zweihander_macos.hex.
5. from ~/Downloads, copy ergodox_ez_base_zweihander_macos 2.hex to zweihander.
6. run diff -u ergodox_ez_base_zweihander_macos.hex ergodox_ez_base_zweihander_macos\ 2.hex
7. notice the two files don't match exactly.

Here's the diff:

--- ergodox_ez_base_zweihander_macos.hex    2024-06-16 01:29:28
+++ ergodox_ez_base_zweihander_macos 2.hex  2024-06-16 08:15:08
@@ -486,8 +486,8 @@
 :101E50007F01812F0E9477111F5F1E3009F441C05E
 :101E60000817A1F38091BE0181FD0E947D110E949F
 :101E7000DB0EC82E10E099249394812F0E947711D5
-:101E8000082FF50181915F01081729F1D02ED8267E
-:101E9000F12CEE24E3948D2D8E21A9F0CC2099F025
+:101E8000082FF50181915F01081729F1F12CEE244B
+:101E9000E394D02ED8268D2D8E21A9F0CC2099F058
 :101EA000F9821A830E9441159C838B839D8291E065
 :101EB000802F8E2109F490E049815A816B817C81C9
 :101EC0008D810E944F05F394EE0CF6E0FF12E3CFF4

A bit more graphically, courtesy of diff-so-fancy:

I don't expect to be The Guy Who Caught The Next Jia Tan, but I would expect these two generated .hex files to be identical. It's not like there's highly-optimized floating-point code in there, right?

Keyboard Used

ergodox_ez/base

Link to product page (if applicable)

No response

Operating System

macOS

qmk doctor Output

Ψ QMK Doctor is checking your environment.
Ψ CLI version: 1.1.5
Ψ QMK home: /Users/comatoast/Projects/Others/qmk_firmware
Ψ Detected macOS 14.5 (Apple Silicon).
Ψ Userspace enabled: False
Ψ Git branch: master
Ψ Repo version: 0.25.8
⚠ The official repository does not seem to be configured as git remote "upstream".
Ψ CLI installed in virtualenv.
Ψ All dependencies are installed.
Ψ Found arm-none-eabi-gcc version 14.1.0
Ψ Found avr-gcc version 8.5.0
Ψ Found avrdude version 7.3
Ψ Found dfu-programmer version 1.1.0
Ψ Found dfu-util version 0.11
Ψ Submodules are up to date.
Ψ Submodule status:
Ψ - lib/chibios: 2024-02-17 19:20:06 +0000 --  (be44b3305)
Ψ - lib/chibios-contrib: 2024-04-03 20:39:24 +0800 --  (77cb0a4f)
Ψ - lib/googletest: 2021-06-11 06:37:43 -0700 --  (e2239ee6)
Ψ - lib/lufa: 2022-08-26 12:09:55 +1000 --  (549b97320)
Ψ - lib/vusb: 2022-06-13 09:18:17 +1000 --  (819dbc1)
Ψ - lib/printf: 2022-06-29 23:59:58 +0300 --  (c2e3b4e)
Ψ - lib/pico-sdk: 2023-02-12 20:19:37 +0100 --  (a3398d8)
Ψ - lib/lvgl: 2022-04-11 04:44:53 -0600 --  (e19410f8)
Ψ QMK is ready to go, but minor problems were found

### Is AutoHotKey / Karabiner installed

- [ ] AutoHotKey (Windows)
- [ ] Karabiner (macOS)

### Other keyboard-related software installed

_No response_

### Additional Context

_No response_

[tzarc]

You’d need to at minimum compile with -e VIA_ENABLE=no -e COMMAND_ENABLE=no -e SKIP_VERSION=yes in order to prevent any timestamps or other temporal information such as git revision from being included in the final binary.

you’d also need to ensure you’re building from the exact same environment — GHA has one version of GCC, your build env has another.

[tzarc]

For reference, this script is what I use for validating like-for-like PRs: https://github.com/tzarc/qmk_userspace/blob/main/bin/pr-sha1-verification.sh

It intentionally disables all time/revision based mutations and performs builds, ensuring final binaries have matching SHA1s. Basically reproducible builds works fine so long as you have an identical environment — even the same version of GCC can output different binaries based on the configuration options given to it when building the toolchain itself.

[sigprof]

You are really lucky to get such a small difference between the compiled binaries after using different versions of the toolchain. Basically, the compiler decided to do some operations in a different order when optimizing the code, which in the end gives the same result (this disassembly matches one of your binaries, while the other has the 2 first instructions moved after the next 3 ones):

00001e8c <.Loc.176>:
…/qmk_firmware/quantum/keyboard.c:560
    1e8c:       d0 2e           mov     r13, r16
    1e8e:       d8 26           eor     r13, r24

00001e90 <.LBB25>:
…/qmk_firmware/quantum/keyboard.c:567
    1e90:       f1 2c           mov     r15, r1

00001e92 <.LBE25>:
…/qmk_firmware/quantum/keyboard.c:566
    1e92:       ee 24           eor     r14, r14
    1e94:       e3 94           inc     r14

If you use Docker to build, so that you would use the same container image as the GitHub workflow, you would have a higher chance to get an identical firmware image (but note that in some cases, like the ones mentioned in the previous comments, the firmware image actually embeds some timestamps on purpose, so even rebuilding the exact same code using the exact same toolchain may give different results).

[adiabatic]

Good to know that it's almost certainly nothing to raise an eyebrow about.

Thanks, everyone!