Jenkins agent exits immediately without any logs and error code 159 (RPI 4, aarch64, Raspbian 64 bit)

[stad-nico]

Jenkins and plugins versions report

Environment

```text Jenkins: 2.440.2 OS: Linux - 6.1.21-v8+ Java: 17.0.10 - Eclipse Adoptium (OpenJDK 64-Bit Server VM) --- ant:497.v94e7d9fffa_b_9 antisamy-markup-formatter:162.v0e6ec0fcfcf6 apache-httpcomponents-client-4-api:4.5.14-208.v438351942757 asm-api:9.7-33.v4d23ef79fcc8 bootstrap5-api:5.3.3-1 bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9 branch-api:2.1152.v6f101e97dd77 build-timeout:1.32 caffeine-api:3.1.8-133.v17b_1ff2e0599 checks-api:2.2.0 cloudbees-folder:6.901.vb_4c7a_da_75da_3 commons-lang3-api:3.13.0-62.v7d18e55f51e2 commons-text-api:1.11.0-95.v22a_d30ee5d36 credentials:1337.v60b_d7b_c7b_c9f credentials-binding:657.v2b_19db_7d6e6d dark-theme:439.vdef09f81f85e display-url-api:2.200.vb_9327d658781 durable-task:550.v0930093c4b_a_6 echarts-api:5.5.0-1 email-ext:2.105 font-awesome-api:6.5.1-3 git:5.2.1 git-client:4.7.0 github:1.38.0 github-api:1.318-461.v7a_c09c9fa_d63 github-branch-source:1785.v99802b_69816c gradle:2.10 gson-api:2.10.1-15.v0d99f670e0a_7 instance-identity:185.v303dc7c645f9 ionicons-api:70.v2959a_b_74e3cf jackson2-api:2.17.0-379.v02de8ec9f64c jakarta-activation-api:2.1.3-1 jakarta-mail-api:2.1.3-1 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.9-1 jjwt-api:0.11.5-112.ve82dfb_224b_a_d joda-time-api:2.12.7-29.v5a_b_e3a_82269a_ jquery3-api:3.7.1-2 json-api:20240303-41.v94e11e6de726 json-path-api:2.9.0-58.v62e3e85b_a_655 junit:1265.v65b_14fa_f12f0 ldap:719.vcb_d039b_77d0d mailer:472.vf7c289a_4b_420 matrix-auth:3.2.2 matrix-project:822.824.v14451b_c0fd42 mina-sshd-api-common:2.12.1-101.v85b_e08b_780dd mina-sshd-api-core:2.12.1-101.v85b_e08b_780dd okhttp-api:4.11.0-172.vda_da_1feeb_c6e pam-auth:1.10 pipeline-build-step:540.vb_e8849e1a_b_d8 pipeline-github-lib:42.v0739460cda_c4 pipeline-graph-analysis:216.vfd8b_ece330ca_ pipeline-groovy-lib:704.vc58b_8890a_384 pipeline-input-step:491.vb_07d21da_1a_fb_ pipeline-milestone-step:119.vdfdc43fc3b_9a_ pipeline-model-api:2.2184.v0b_358b_953e69 pipeline-model-definition:2.2184.v0b_358b_953e69 pipeline-model-extensions:2.2184.v0b_358b_953e69 pipeline-rest-api:2.34 pipeline-stage-step:312.v8cd10304c27a_ pipeline-stage-tags-metadata:2.2184.v0b_358b_953e69 pipeline-stage-view:2.34 plain-credentials:179.vc5cb_98f6db_38 plugin-util-api:4.1.0 resource-disposer:0.23 scm-api:689.v237b_6d3a_ef7f script-security:1335.vf07d9ce377a_e snakeyaml-api:2.2-111.vc6598e30cc65 ssh-credentials:337.v395d2403ccd4 ssh-slaves:2.948.vb_8050d697fec structs:337.v1b_04ea_4df7c8 theme-manager:215.vc1ff18d67920 timestamper:1.26 token-macro:400.v35420b_922dcb_ trilead-api:2.142.v748523a_76693 variant:60.v7290fc0eb_b_cd workflow-aggregator:596.v8c21c963d92d workflow-api:1291.v51fd2a_625da_7 workflow-basic-steps:1049.v257a_e6b_30fb_d workflow-cps:3894.vd0f0248b_a_fc4 workflow-durable-task-step:1336.v768003e07199 workflow-job:1400.v7fd111b_ec82f workflow-multibranch:773.vc4fe1378f1d5 workflow-scm-step:427.v4ca_6512e7df1 workflow-step-api:657.v03b_e8115821b_ workflow-support:896.v175a_a_9c5b_78f ws-cleanup:0.45 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Raspbian 64 Bit on Raspberry Pi 4 (aarch64)

Reproduction steps

Followed the exact same steps as described in the docker hub overview, meaning:

1. Create new node in jenkins
2. start the agent with the command docker run --platform linux/arm64 --init jenkins/inbound-agent -url http://jenkins-server:port <secret> <agent name> with the correct port (50000), secret and agent name.

The platform was needed because otherwise docker would complain about the mismatch between the requested image's platform (linux/arm64) and the detected host platform (linux/arm/v7).

Expected Results

Ideally logs showing that the agents startup was successfully and the connection to the master was established, or even any error logs showing what went wrong.

Actual Results

No logs or errors messages. Running the command simply exited back to the command line after few milliseconds.

Anything else?

No response

Are you interested in contributing a fix?

No response

[MarkEWaite]

If you're running the Pi-4 as an arm/v7 architecture (probably a 32 bit operating system), then I don't think that you can expect that an aarch64 container image will run. I believe that you'll need to run the Pi-4 as an aarch64 architecture (a 64 bit operating system) in order to run a 64 bit container image.

[MarkEWaite]

You build your own container image for armv7 based on the work that @gounthar did in:

• 778

[stad-nico]

I am running the pi with 64 bit on aarch64. uname -m yields aarch64

[MarkEWaite]

Are you sure that you're running a docker version that is built for aarch64?

[MarkEWaite]

Maybe this page offers some additional help?

[gounthar]

What I find puzzling is the armv7 detection.

So, uname -m gives you aarch64? The Raspberry Pi 4B comes with an Armv8-A processor. Armv8-A supports both armv7 and aarch64 instruction sets...

If you've installed a 32 bits version of Docker because of a configuration mismatch, the image won't work for you. Did you use the standard package manager installation, or did you use the Docker utility script?

[stad-nico]

So, uname -m gives you aarch64?

Yes, it does.

If you've installed a 32 bits version of Docker because of a configuration mismatch, the image won't work for you. Did you use the standard package manager installation, or did you use the Docker [utility script] (https://docs.docker.com/engine/install/debian/#install-using-the-convenience-script)?

I installed with the standard package manager, however I am not sure if my docker installation is 32 bit or 64 bit... How would I check this? docker version prints:

Client:
 Version:           20.10.5+dfsg1
 API version:       1.41
 Go version:        go1.15.15
 Git commit:        55c4c88
 Built:             Mon May 30 18:34:49 2022
 OS/Arch:           linux/arm
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.5+dfsg1
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.15.15
  Git commit:       363e9a8
  Built:            Mon May 30 18:34:49 2022
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          1.4.13~ds1
  GitCommit:        1.4.13~ds1-1~deb11u4
 runc:
  Version:          1.0.0~rc93+ds1
  GitCommit:        1.0.0~rc93+ds1-5+deb11u3
 docker-init:
  Version:          0.19.0
  GitCommit:

Interestingly, using dpkg --print-architecture does say armhf indicating a 32 bit OS (though I think to remember installing 64 bit). I am a bit onfused about this mismatch between uname -m and dpkg --print-architecture... Shouldn`t both say 64 bit if I run a 64 bit OS?

getconf LONG_BIT also says 32...

[gounthar]

The uname -m command provides information about the machine hardware name, which is typically related to the architecture of the CPU (aarch64 in your case). However, it's important to note that this command actually returns the architecture for which the kernel was compiled, not the CPU itself. For instance, if you're running a 32-bit kernel on a 64-bit capable CPU, uname -m would likely return a value indicating a 32-bit system. This is because the kernel is interfacing with the hardware and the command is reporting on the kernel's perspective of the system. So... You may have a 64 bits Kernel on a 64 bits machine, with 32 bits packages. On a full aarch64 machine, you would have gotten :

OS/Arch: linux/arm64

I guess it would be better to uninstall all the docker-related packages, and reinstall with the utility script.

[stad-nico]

Thank you for this detailed information. I reinstalled docker using the utility script, but it still says OS/Arch: linux/arm and docker run still does nothing.

[gounthar]

Did you uninstall before reinstalling? If so, you may have a flaky system detected as arm32. I guess docker run --rm busybox uname -m will give you armv7l.

[stad-nico]

I did uninstall it before reinstalling.

I guess docker run --rm busybox uname -m will give you armv7l.

It does not, it says aarch64

[gounthar]

Interesting, then... 🤔 Are you still using the --platform linux/arm64 option?

[stad-nico]

Yes, I do. Strangely I have no problems with other docker images, pihole, traefik, wireguard etc all run fine with the linux/arm64 arch. Even the jenkins server image runs fine. This image is the first that raised this problem.

[gounthar]

Let's have a look at the architecture of the image you're using.

# Create a new container (without starting it) and store its ID
container_id=$(docker container create jenkins/inbound-agent)

# Get the Image ID from the container
image_id=$(docker inspect --format '{{.Image}}' $container_id)

# Use the Image ID to get the architecture of the image
docker image inspect --format '{{.Architecture}}' $image_id

It returns arm64 on my machine.

[stad-nico]

Same for me (arm64), however I get the warning The requested image's platform (linux/arm64) does not match the detected host platform (linux/arm/v8) and no specific platform was requested

[gounthar]

What if you remove the --platform option then? 🤔

[stad-nico]

If I omit the --platform option it still gives me the warning and exits immediately.

[gounthar]

Very strange (to me, at least). I just tried on an armv7 machine, and got arm. 🤔 What is your version of docker now? The one you had when you opened this issue was kind of old, but I guess you have the most recent one now.

[stad-nico]

Client: Docker Engine - Community
 Version:           26.0.0
 API version:       1.45
 Go version:        go1.21.8
 Git commit:        2ae903e
 Built:             Wed Mar 20 15:18:13 2024
 OS/Arch:           linux/arm
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.0.0
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.8
  Git commit:       8b79278
  Built:            Wed Mar 20 15:18:13 2024
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Does yours say OS/Arch: linux/arm64?

[gounthar]

Yes it does.

docker version
Client: Docker Engine - Community
 Version:           26.0.0
 API version:       1.45
 Go version:        go1.21.8
 Git commit:        2ae903e
 Built:             Wed Mar 20 15:18:14 2024
 OS/Arch:           linux/arm64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.0.0
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.8
  Git commit:       8b79278
  Built:            Wed Mar 20 15:18:14 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Something's fishy on your installation.

[stad-nico]

Hm, this is odd. Is there any way to force docker to install to linux/arm64? docker info says Architecture: aarch64...

[gounthar]

One way to force the installation of the 64-bit version of Docker is to manually download and install the Docker package for arm64. Here's how you can do it: First, uninstall the current Docker version: for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc docker-ce; do sudo apt-get remove $pkg;sudo apt-get purge $pkg; done Then, download the Docker package for arm64: wget https://download.docker.com/linux/debian/dists/buster/pool/stable/arm64/docker-ce_26.0.0-1~debian.10~buster_arm64.deb Install the downloaded package: sudo dpkg -i docker-ce_26.0.0-1~debian.10~buster_arm64.deb Verify the Docker installation: docker version

🤞

And on a side note: https://forums.raspberrypi.com/viewtopic.php?p=2091536&sid=b31cf90cebb33ea3aa76e17b155812dd#p2091536

[stad-nico]

Install the downloaded package: sudo dpkg -i docker-ce_26.0.0-1~debian.10~buster_arm64.deb

This fails saying package architecture (arm64) does not match system (armhf). using --force architecture fails with Errors were encountered while processing: docker-ce:arm64.

armhf matches the output of dpkg --print-architecture, however dpkg --print-foreign-architectures yields aarch64 as expected. See this

And on a side note: https://forums.raspberrypi.com/viewtopic.php?p=2091536&sid=b31cf90cebb33ea3aa76e17b155812dd#p2091536

This unfortunately did not work for me.

[gounthar]

It looks like you have an interesting workaround in the link you gave.

[stad-nico]

It would be if it would work😁I dont seem to get this fixed today, maybe I have the energy to try this on a fresh Raspbian 64 Bit OS tomorrow where the different arch commands yield more consistent results. However this is my first time taking such a deep dive into the different architectures and differences between kernel, cpu and os architecture so I dont quite know how to interpret these results. Anyway, I will post my results hopefully tomorrow.

[stad-nico]

Well, here's my update: With a fresh Raspbian installation (64 bit) everything works as expected and starting the agent is as straightforward as it should be. This time, dpkg --print-architecture prints the expected aarch64 instead of armhf and the docker utility script installs correctly to the arch linux/arm64. This way I was also able to omit the --platform flag because docker correctly identified the architecture.

However, I was not able to resolve this problem on my other (probably broken/misconfigured) sd card, where dpkg --print-architecture prints the unexpected armhf. For some users this may be resolved by adding arm_64bit=1 to the config file located either at /boot/config.txt or /boot/firmware/config.txt. This ensures to use a 64 bit kernel (if available) and should make dpkg --print-architecture print aarch64. This way docker should also install correctly to linux/arm64 and everything should work as expected.

I was not able to reproduce this and thus not able to confirm it, but for some it may work even though not recommended.

[gounthar]

Thanks a lot for your feedback. 🙏 I'm glad to hear you finally made it. 🚀