Packed Windows amd64 Go binary crashes under arm64 emulation

[ViRb3]

Issue description

I have a MacBook Pro M1 Max running Windows 11 arm64 under Parallels. I tried compressing an open-source amd64 Windows binary compiled using Go:

• https://github.com/gravitl/netmaker/releases/download/v0.14.6/netclient.exe

I get a nicely compressed binary at ~30% of the original size. The packed binary runs great under a real amd64 Windows machine, but it crashes very strangely under arm64 emulation:

panic: regexp: Compile(`^[\p{L}]+$`): error parsing regexp: invalid character class range: `\p{L}`

goroutine 1 [running]:
regexp.MustCompile({0x7ff72b8adf23, 0xa})
        C:/Program Files/Go/src/regexp/regexp.go:317 +0xbb
github.com/go-playground/validator/v10.init()
        C:/Users/alexf/go/pkg/mod/github.com/go-playground/validator/v10@v10.11.0/regexes.go:72 +0x79f

I even tried packing with all options disabled, but that did not have an effect:

.\upx.exe --strip-relocs=0  --compress-resources=0 --compress-icons=0 --compress-exports=0  .\netclient.exe

The unpacked binary runs great under emulation, as expected. Other packed Go binaries run fine on the same system, so I'm not sure what's different here.

Environment

• UPX version used (upx --version):

  upx 3.96
  NRV data compression library 0.84
  UCL data compression library 1.03
  zlib data compression library 1.2.3
  LZMA SDK version 4.43

• Host Operating System and version: Windows 11 arm64 22H2 22622.450 (ran under Parallels 18 on MacBook Pro M1 Max)
• Host CPU architecture: arm64
• Target Operating System and version: arm64
• Target CPU architecture: amd64

[ViRb3]

@jreiser may I know why this is not considered a bug?

[jreiser]

@ViRb3 This is not considered to be a bug in UPX because:The packed binary runs great under a real amd64 Windows. Therefore the bug is somewhere in the emulation stack: either in emulating x86_64 instructions on aarch64 hardware, and/or in emulating calls on Windows API via calls on MacOS API. So the bug lies with Parallels and/or (Apple) Rosetta.

[ViRb3]

I see. If I had to make an educated guess, I am willing to bet that the issue is not in Parallels or Rosetta (which is not used here), but I do not have the time or interest to dive into this more. Parallels is based on Apple's own hypervisor, and it runs real Windows, the same way VirtualBox, VMWare or HyperV would run real Windows. In my opinion it is naive to assume that a hypervisor would make an error that would run entire operating systems with all of their software and hardware stack, but fail on something like upx. I suspect the problem is in the way that Windows for ARM itself emulates amd64 to arm64. More specifically, I doubt it is even incorrect emulation, but more an internal call or structure that behaves differently (not incorrectly) under emulation. Thus, I believe that a fix is feasible, and will likely boil down to something very simple. Unfortunately, as mentioned before, I do not have the time to dive on this, but maybe keep this issue open if somebody has more information or wants to pick it up?

[jreiser]

... an internal call or structure that behaves differently (not incorrectly) under emulation.

If running under emulation does not produce identical results (modulo timing, of course; but including being "bug for bug compatible") with running in the real environment, then the emulator has a bug. If some particular OS call has varying interpretations, then it is the emulator's job to know, and to force the the OS to produce results that are identical to those of the real environment.

[CryptoManiac]

If running under emulation does not produce identical results (modulo timing, of course; but including being "bug for bug compatible") with running in the real environment, then the emulator has a bug. If some particular OS call has varying interpretations, then it is the emulator's job to know, and to force the the OS to produce results that are identical to those of the real environment.

I believe that there is a misunderstanding because hardware virtualization has nothing to do with emulation. I'm not sure what happened in ViRb3's case but Windows running on Parallels is no different from Windows running on Surface X or whatever. There is no "emulation" of anything except for some paravirtualized hardware which has nothing to do with code execution.

[jreiser]

@CryptoManiac The top Description [edited, but we must assume correctly] says: The packed binary runs great under a real amd64 Windows machine, but it crashes very strangely under arm64 emulation: "Packed binary runs great under a real amd64 Windows" means that the output from UPX after compressing netclient.exe, a 64-bit amd64 (Intel) PE file, runs correctly on 64-bit Intel hardware. This agrees with:

$ file netclient.exe
netclient.exe: PE32+ executable (console) x86-64, for MS Windows

The top Description also says that the Host and Target are arm64, which is ARM aarch64 (not Intel). So the original poster is running the compressed Intel binary on an ARM machine. There MUST be instruction-by-instruction emulation somewhere.

Also, the claim hardware virtualization has nothing to do with emulation is another misunderstanding. Hardware virtualization is an emulation of the hardware using [usually] the same kind of hardware, often for the purpose of providing multiple simultaneous independent operating system environments using only one machine. Typically the emulation is somewhat "thin": intercept the trap which occurs for supervisor mode instructions, and emulate them; just use the real hardware for any non-privileged instructions.

So the take-away is: UPX-compressed 64-bit Windows executable for Intel hardware runs on Intel hardware, but not on ARM hardware. Whatever is providing the ability to "run Intel executables on ARM hardware" has a bug.

[ViRb3]

If I had to take a very wild guess, it's probably UPX assuming some internal structure which has changed under arm64 emulation of x86_64. Is this a bug in the emulation? It depends on whether you consider accurate replication of internal structures in scope of emulation. As far as I know, they are never supposed to be stable, and such changes are fair play as they would not break regular apps.

Anyway, I only wanted to get this issue to the attention of the UPX maintainers. It is at this point more than clear that there is no interest in fixing or working it around, so I am happy to leave it here.