Build failing on balena for RPi 4 8GB RAM

[ketilmo]

Hi,

Thanks for creating a great setup. I have been running basicstation-docker with a balena for a while. From commit 4ce007a6fed6fa7781af92b044bad1e1b5e78aae and onwards, the build process has started failing on my Raspberry Pi 4 with 8 GB RAM. From the logs:

Applying basicstation.v2.0.6.patch ...
../patches/basicstation.v2.0.6.patch:16: trailing whitespace.
../patches/basicstation.v2.0.6.patch:23: trailing whitespace.
../patches/basicstation.v2.0.6.patch:44: trailing whitespace.
../patches/basicstation.v2.0.6.patch:50: trailing whitespace.
../patches/basicstation.v2.0.6.patch:298: trailing whitespace.

warning: squelched 9 whitespace errors
warning: 14 lines add whitespace errors.

-------------------------------------------------------
Building BasicStation for v2
-------------------------------------------------------
setup.gmk:73: No toolchain for platform 'rpi' and local arch is not 'arm-linux-gnueabihf'

mkdir -p build-local/bin
gcc -std=gnu11 -Isrc -DCFG_prog_crc32 src/crc32.c -o build-local/bin/crc32
mkdir -p build-local/bin
gcc -std=gnu11 -Isrc -DCFG_prog_genkwcrcs src/genkwcrcs.c -o build-local/bin/genkwcrcs
build-local/bin/genkwcrcs $(cat src/kwlist.txt | sed -e '/^#/d;s/[ \t]\+#.*//') > build-local/temp-kwcrc.h
mv build-local/temp-kwcrc.h src/kwcrc.h
platform=rpi variant=std make -C deps/mbedtls
make[1]: Entering directory '/app/basicstation/deps/mbedtls'
../../setup.gmk:73: No toolchain for platform 'rpi' and local arch is not 'arm-linux-gnueabihf'

platform=rpi variant=std ./prep.sh
Cloning into 'git-repo'...

Note: checking out '8b3f26a5ac38d4fdccbc5c5366229f3e01dafcc0'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
  git checkout -b <new-branch-name>

Cloning into 'platform-rpi'...

Note: checking out '8b3f26a5ac38d4fdccbc5c5366229f3e01dafcc0'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
  git checkout -b <new-branch-name>

HEAD is now at 8b3f26a Merge pull request #868 from ARMmbed/mbedtls-2.28.0rc0-pr
make[2]: Entering directory '/app/basicstation/deps/mbedtls/platform-rpi'
make[3]: Entering directory '/app/basicstation/deps/mbedtls/platform-rpi/library'
make[3]: Leaving directory '/app/basicstation/deps/mbedtls/platform-rpi/library'
make[3]: Entering directory '/app/basicstation/deps/mbedtls/platform-rpi/programs'
make[4]: Entering directory '/app/basicstation/deps/mbedtls/platform-rpi/programs/fuzz'
make[4]: Leaving directory '/app/basicstation/deps/mbedtls/platform-rpi/programs/fuzz'
make[3]: Leaving directory '/app/basicstation/deps/mbedtls/platform-rpi/programs'
make[3]: Entering directory '/app/basicstation/deps/mbedtls/platform-rpi/tests'
make[3]: Leaving directory '/app/basicstation/deps/mbedtls/platform-rpi/tests'
make[2]: Leaving directory '/app/basicstation/deps/mbedtls/platform-rpi'
make --no-print-directory all platform=rpi variant=std
../../setup.gmk:73: No toolchain for platform 'rpi' and local arch is not 'arm-linux-gnueabihf'

make -C platform-rpi lib CC="NO-TOOLCHAIN-FOUND-gcc" AR="NO-TOOLCHAIN-FOUND-ar" LD="NO-TOOLCHAIN-FOUND-ld" CFLAGS="-Wall -std=gnu11"
  CC    aes.c
make[4]: NO-TOOLCHAIN-FOUND-gcc: Command not found

make[4]: *** [Makefile:284: aes.o] Error 127

make[3]: *** [Makefile:17: lib] Error 2

make[2]: *** [makefile:43: platform-rpi/library/libmbedtls.a] Error 2

make[1]: *** [makefile:56: platform-rpi] Error 2

make[1]: Leaving directory '/app/basicstation/deps/mbedtls'
make: *** [makefile:67: deps/mbedtls] Error 2

For commit 1f62150768b9973af7e01073f1dc07d35994abd8, everything works fine.

I've done a little bit of research and see that the same errors were mentioned in the main basicstation repo a while ago: https://github.com/lorabasics/basicstation/issues/171

Any assistance would be highly appreciated.

Kind regards, Ketil

[ketilmo]

Same issue confirmed on a Raspberry Pi 4 with 4 GB RAM.

[ketilmo]

Also failing on a Raspberry Pi 3 with 1 GB RAM, but with a different error message:

Setting up libjq1:armhf (1.5+dfsg-2+b1) ...
Setting up jq (1.5+dfsg-2+b1) ...
Setting up gpiod (1.2-3) ...
Processing triggers for libc-bin (2.28-10+deb10u2) ...
Removing intermediate container e3ffa6b61807
---> 8b782129cc9d
Step 28/36 : WORKDIR /app
---> Running in 8dac839a656a
Removing intermediate container 8dac839a656a
---> 9f5e3c72faed
Step 29/36 : RUN mkdir ./artifacts
---> Running in 393e2a1c6687
Removing intermediate container 393e2a1c6687
---> 1c7fb05ebbe1
Step 30/36 : COPY --from=builder /app/basicstation/build-rpi-${VARIANT}/bin ./artifacts/v2
COPY failed: stat app/basicstation/build-rpi-/bin: file does not exist

[xoseperez]

Hi @ketilmo May I know how are you building the image? I have tried to reproduce the error without success, I can get the image built on a Raspberry Pi 4 without problem.

The patch linked from the basicstation repo was applied a long time ago. As you can see here the image is using the aarch64-linux-gnu to build the rpi model (V2) on an ARM64 machine:

https://github.com/xoseperez/basicstation-docker/blob/ba5fe1b3247049cf8bc6b29a22f62dac98490180/builder/patches/basicstation.v2.0.6.patch#L849)

[ketilmo]

Thanks for your prompt reply, @xoseperez. I appreciate your help.

Yes, it's strange that the build fails between the two mentioned commits. I've been unable to pinpoint any specific differences that would cause this.

I'm building the images using the balena pushcommand.

The good news is that your prebuilt containers work flawlessly, so I have switched to them for now.

[xoseperez]

Balena uses a different Dockerfile and it was not passing the right architecture to the builder. Pushed a fix to the repo. You can try it now.

[ketilmo]

Thanks for your support, @xoseperez!

The process goes further now, but fails at this stage:

Setting up libjq1:arm64 (1.5+dfsg-2+b1) ...
Setting up jq (1.5+dfsg-2+b1) ...
Setting up gpiod (1.2-3) ...
Processing triggers for libc-bin (2.28-10+deb10u2) ...
Removing intermediate container 3a0f1a9085a4
 ---> 3f974859e6d3
Step 28/36 : WORKDIR /app
 ---> Running in bb05c72cf3b7
Removing intermediate container bb05c72cf3b7
 ---> 9fa20ed8985a
Step 29/36 : RUN mkdir ./artifacts
 ---> Running in 072ddbc83431
Removing intermediate container 072ddbc83431
 ---> 4155a6ae9c5c
Step 30/36 : COPY --from=builder /app/basicstation/build-rpi-${
COPY failed: stat app/basicstation/build-rpi-/bin: file does not exist

[xoseperez]

Added the required build variables at the top of the balena Dockerfile.template. Mind I had to remove the docker-compose.yml file in order to force Balena to build the image from the Dockerfile. The solution is not pretty, I don't like it, but it's the best I have found so far. I don't know how to provide build settings from the balena.yml file or if it's even possible.

[ketilmo]

Thank you so much, @xoseperez! It worked perfectly on all the target versions listed in the ticket above. I did not have to delete anything when doing the balena push. Everything worked straight out of the box. Thanks again!

[xoseperez]

Great!