search

nodejs / docker-node

Official Docker Image for Node.js :whale: :turtle: :rocket:
https://hub.docker.com/_/node/
MIT License
8.23k stars 1.97k forks source link

Docker build with GitHub Actions failed on multi arch with node:alpine #1798

Open manuc66 opened 1 year ago

manuc66 commented 1 year ago

Environment

Expected Behavior

The build is expected to complete without error, see: https://github.com/manuc66/node-hp-scan-to/actions/runs/3278864778/jobs/5397750728

Current Behavior

The build fail due to a timeout, see : https://github.com/manuc66/node-hp-scan-to/actions/runs/3304345713/jobs/5453599065

#29 [linux/arm/v7 build 4/4] RUN yarn install -d  && yarn build  && rm dist/*.d.ts dist/*.js.map
Error: The operation was canceled.

Possible Solution

I switched back to node:18-alpine and the build is working again, see: https://github.com/manuc66/node-hp-scan-to/actions/runs/3356995905/jobs/5562501857

stumpylog commented 1 year ago

This also looks to be affecting another project: https://github.com/benphelps/homepage/actions/runs/3382149679

Going from 

-FROM node:current-alpine
+FROM docker.io/node:18-alpine

fixes it. Hard for me to tell if this is a problem with the image or Node at this point.

n-thumann commented 1 year ago

I've stumbled upon the same issue in https://github.com/n-thumann/IPTV-ReStream/pull/188. It can be easily reproduced locally using docker run --rm arm32v7/node:19-alpine node -h. Simply printing the help hangs indefinitely for both arm32v6 and arm32v7, while any other architecture or prior Node.js release prints the help immediately.

This may be an issue related to / in qemu, because running the same command on a natively on Alpine @ armv7 (Raspberry Pi in this case) doesn't cause any issues:

alpine:/tmp# uname -m
armv7l
alpine:/tmp# ./node -v
v19.0.0
alpine:/tmp# ./node -h
Usage: node [options] [ script.js ] [arguments]
[... snip ...]

I also tried to strace and noticed that node -h yields a ton of [pid 52] mremap(0x8080c000, 4096, 8192, 0) = -1 ENOMEM (Out of memory) while hanging. Maybe someone else has an idea what to do here 🤔

tyranron commented 1 year ago

This is the same issue as https://github.com/nodejs/docker-node/issues/1794.

On Node 18 everything is OK, on Node 19 (latest) npm sticks on some architectures. Seems to be upstream issue: https://github.com/npm/cli/issues/5743

tyranron commented 1 year ago

Still reproduces on Node v20 images.

n-thumann commented 1 year ago

I did some investigation and it looks like this is caused by qemu, which is used internally in Docker to emulate foreign architectures, e.g. when running a ARMv7 binary on a x86_64 host. This can be reproduced via:

➜  ~ docker run -it --rm --platform linux/arm/v7 alpine:3.18.2 sh
/ # apk add nodejs-current=20.5.0-r0
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/armv7/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/armv7/APKINDEX.tar.gz
(1/11) Installing ca-certificates (20230506-r0)
(2/11) Installing libgcc (12.2.1_git20220924-r10)
(3/11) Installing libstdc++ (12.2.1_git20220924-r10)
(4/11) Installing ada-libs (2.5.1-r0)
(5/11) Installing brotli-libs (1.0.9-r14)
(6/11) Installing c-ares (1.19.1-r0)
(7/11) Installing icu-data-en (73.2-r2)
Executing icu-data-en-73.2-r2.post-install
*
* If you need ICU with non-English locales and legacy charset support, install
* package icu-data-full.
*
(8/11) Installing icu-libs (73.2-r2)
(9/11) Installing nghttp2-libs (1.55.1-r0)
(10/11) Installing libuv (1.44.2-r2)
(11/11) Installing nodejs-current (20.5.0-r0)
Executing busybox-1.36.1-r0.trigger
Executing ca-certificates-20230506-r0.trigger
OK: 53 MiB in 26 packages
/ # wget https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/qemu-arm-8.0.3-r1.apk
Connecting to dl-cdn.alpinelinux.org (146.75.118.132:443)
saving to 'qemu-arm-8.0.3-r1.apk'
qemu-arm-8.0.3-r1.ap 100% |*******************************************************************************************************************************************************| 1218k  0:00:00 ETA
'qemu-arm-8.0.3-r1.apk' saved
/ #
/ # apk add --allow-untrusted qemu-arm-8.0.3-r1.apk
(1/1) Installing qemu-arm (8.0.3-r1)
Executing busybox-1.36.1-r0.trigger
OK: 56 MiB in 27 packages
/ # qemu-arm -version
qemu-arm version 8.0.3
Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers
/ # /usr/bin/node -v
v20.5.0
/ # qemu-arm -strace /usr/bin/node -h
[...]
55 open("/proc/meminfo",O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 17
55 fcntl64(17,F_SETFD,1) = 0
55 read(17,0x3fffeb2c,4095) = 1363
55 close(17) = 0
55 mremap(1073733632,4096,8192,0,0,4096) = -1 errno=12 (Out of memory)
55 mremap(1073729536,4096,8192,0,0,4096) = -1 errno=12 (Out of memory)
55 mremap(1073725440,4096,8192,0,0,4096) = -1 errno=12 (Out of memory)
55 mremap(1073721344,4096,8192,0,0,4096) = -1 errno=12 (Out of memory)
55 mremap(1073717248,4096,8192,0,0,4096) = -1 errno=12 (Out of memory)
55 mremap(1073713152,4096,8192,0,0,4096) = -1 errno=12 (Out of memory)
55 mremap(1073709056,4096,8192,0,0,4096) = -1 errno=12 (Out of memory)
55 mremap(1073704960,4096,8192,0,0,4096) = -1 errno=12 (Out of memory)
55 mremap(1073700864,4096,8192,0,0,4096) = -1 errno=12 (Out of memory)
[...]
n-thumann commented 1 year ago

This seems to be indeed caused by a bug in qemu and is tracked in https://gitlab.com/qemu-project/qemu/-/issues/1729.

agners commented 8 months ago

It seems that this is resolved with QEMU 8.1.0. There are still many mremap syscalls, but the same commands above complete when using https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/qemu-arm-8.1.3-r1.apk.

tyranron commented 8 months ago

With https://github.com/tonistiigi/binfmt/pull/144 (QEMU 8.1.4) and node:21 it still doesn't work for me on arm32v6, arm32v7 and s390x platforms. Tried building on both macos-latest and ubuntu-latest runners:

fuhlich commented 3 weeks ago

I've tracked the issue some time now. Is it correct, that the root cause can be pinned down to the qemu bug? There does not appear to be anyone taking up the matter.