Linux ARM64 images appear to be x86_64

[clarkezone]

I just attempted to download and run the linux ARM64 images dropped as part of your build process and they appear to be AMD64 as demonstrated here. Same result from 0.7.8 and 0.7.9 builds from your releases page.

This is further verified by the fact that when I deploy an image containing the above binary on a Raspberry Pi 4 I’m getting an exec format error which indicates binary is wrong architecture:

[ganigeorgiev]

@clarkezone The releases are generated by the following goreleaser config - https://github.com/pocketbase/pocketbase/blob/master/.goreleaser.yaml.

I've just downloaded the latest arm64 build and tried to run it on my local machine (linux x86_64) and I get, as expected, "cannot execute binary file: Exec format error".

Additionally, this is the result of file pocketbase:

pocketbase: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, Go BuildID=....

[ganigeorgiev]

I've just checked both v0.7.8 and v0.7.9 releases and the file command for both binaries indicates that they are arm64.

The size of the bundles for the 2 architectures is also different (for amd64 is ~37.5mb and for arm64 is ~36.2MB), so we definitely don't ship the same binaries.

Unfortunately, I don't have an ARM device to further verify it, but the output of the file cmd should be correct.

[clarkezone]

OK.. I see the same behavior as above but the binary runs fine on amd64 and not arm64 so there is clearly something wrong. I'll try building it myself and see if i can figure it out

[ganigeorgiev]

I'm not able to run the arm64 executable on my fedora x86_64 machine (I've also tested in a different ubuntu vm).

To build the base app example:

1. clone the repo
2. navigate to examples/base
3. run GOARCH=arm64 GOOS=linux CGO_ENABLED=0 go build main.go

[clarkezone]

Haven't yet got definitively to the bottom of the problem, but I'm seeing:

1. the linux ARM64 binary from your build server does not work on arm64 raspberry pi
2. a binary produced by cross-compiling on a ubuntu x86_64 VM using go version go1.19.1 linux/amd64 does not work on arm64 RaPi
3. a binary produced by cross-compiling on an M1 Mac using go version go1.18.2 darwin/arm64 does work on arm64 RaPi

In all 3 cases above, file pocketbase shows expected ELF 64-bit LSB executable, ARM aarch64

So I'm guessing this is a golang x-compilation bug. To fully root cause would need to try golang 1.18.2 on x64_64 ubuntu and golang 1.19.1 on my M1 Mac which I likely won't have chance to do until the weekend. But at least I'm unblocked now.

[ganigeorgiev]

@clarkezone Thank you for debugging this. That's definitely strange. The only thing outside of a Go regression I could think of is that the produced binaries are not statically linked for some reason, eg. CGO_ENABLED was on during build. Could you confirm that the file command shows that the executable is statically linked? (but then again, I can confirm that the binary from the github action release is statically linked and it still doesn't seem to work...)

I'll reopen the issue for better visibility.

[ganigeorgiev]

Just for info, 2 other users reported in https://github.com/pocketbase/pocketbase/discussions/764 that the arm64 binary works fine on the Oracle arm cloud service.

I'm still not sure how you are able to run the arm64 binary on the x86_64 machine - maybe there is an emulator similar to box86 that is running behind the scene or the issue here is related to something specific to the OS that is installed on the Raspberry Pi?

[KevinSchwuchow]

@clarkezone Hello my best guess why you are able to run arm64 binaries on amd64 is that you have QemuUserEmulation enabled, with it i can reproduce you issue without it i can not.

And i think the reason the it doesnt work on your raspberry pi 4 is that you are using a operation system which doesnt support 64 bit, for me, i previously had problems with this when i was using Raspberry Pi OS (even using the 64 bit version). try using manjaro arm or ubuntu server and see if you still have this problem

[clarkezone]

@KevinSchwuchow my rapi cluster is very definitely running a 64bit ARM OS:

  james@rapi-c2-n1:~$ hostnamectl
   Static hostname: rapi-c2-n1
         Icon name: computer
        Machine ID: 4fd1b1da7ea44136adecb4c264a767d7
           Boot ID: a514d02c35a642b2b0ec30de92f0aa50
  Operating System: Ubuntu 20.04.5 LTS
            Kernel: Linux 5.4.0-1069-raspi
      Architecture: arm64
james@rapi-c2-n1:~$ 

How can I tell if QemuUserEmulation is on on the amd64 box?

[KevinSchwuchow]

@clarkezone i think you can try the following command and see if it contains qemu-aarch64 (which i think is arm64)

ls /proc/sys/fs/binfmt_misc

regarding the 64 bit arm os, could you try to eliminate variables and try to run the following scripting on a newly installed pi? it works on my raspberry pi 4 with manjaro arm, maybe kubernetes is causing the issue here?

wget https://github.com/pocketbase/pocketbase/releases/download/v0.7.9/pocketbase_0.7.9_linux_arm64.zip
wget https://github.com/pocketbase/pocketbase/releases/download/v0.7.9/checksums.txt
sha256sum -c checksums.txt --ignore-missing
unzip pocketbase_0.7.9_linux_arm64.zip
./pocketbase

[haljac]

3. GOARCH=arm64 GOOS=linux CGO_ENABLED=0 go build main.go

@clarkezone I am running a Raspberry Pi 4, and it's important to note that uname -m indicates arm architecture, not arm64. When using the suggested env above, I was unable to get the pocketbase source to compile due to the error that OP specified. However, if you run instead:

GOARCH=arm GOOS=linux CGO_ENABLED=0 go build main.go

... inside of examples/base, you will produce a valid binary. Hope this helps.

[clarkezone]

Thanks I will give that a try, actively upgrading my cluster to pocketbase 0.8 this weekend

[ganigeorgiev]

@jackvhall, @clarkezone

If you are able to run the binary compiled with GOARCH=arm (eg. armv7), then this probably means that your OS is not 64bit. We already have prebuilt arm binary that you could download from the "Assets section" under each release. For v0.8.0 that is https://github.com/pocketbase/pocketbase/releases/download/v0.8.0/pocketbase_0.8.0_linux_armv7.zip

[clarkezone]

The OS is definitely 64bit so that may mean this doesn't work

[ganigeorgiev]

I'm closing this issue since I'm not able to reproduce it.

Several users reported that it works fine for them using the free Oracle Ampere (arm) cloud offerings and a coworker of mine also have tested successfully the arm64 binary on their Raspberry Pi 4 (Ubuntu 22.04).

If anyone is still experiencing the same issue, please provide more details about your device, used OS and the general steps that you've executed, so that I can try to replicate them.

[clarkezone]

Sounds good, I will reopen if this still repos when I get back to this.

My specific issue has been using official binaries on 64bit (ARM64) based Ubuntu OS installs. Are the folks who have it working on Ubuntu using 64bit or 32bit installs?

I am unblocked currently with private binaries and dealing with other stability issues with pb that I'm working on root causing, when done with that will come back to this and reopen if necessary.