Balena crossbuild fails

[danielgratzl]

Actual behavior I'm using a Belena image (balenalib/raspberrypi3-debian-python:3.6.8-build) to build an image containing a Python3 application for rasberry.

Expected behavior Build fails with the following message

/kaniko/executor  --context $CI_PROJECT_DIR/$DOCKER_CONTEXT_BASE_DIR/$DOCKER_CONTEXT  --dockerfile $CI_PROJECT_DIR/$DOCKER_CONTEXT_BASE_DIR/$DOCKER_CONTEXT/Dockerfile  --destination $REGISTRY/$IMAGE_NAME:$CI_COMMIT_TAG
INFO[0000] Using dockerignore file: /builds/geopraevent/services/external/henry/Docker/Production/geobrugg-henry/.dockerignore 
INFO[0000] Retrieving image manifest balenalib/raspberrypi3-debian-python:3.6.8-build 
INFO[0000] Retrieving image balenalib/raspberrypi3-debian-python:3.6.8-build 
INFO[0001] Retrieving image manifest balenalib/raspberrypi3-debian-python:3.6.8-build 
INFO[0001] Retrieving image balenalib/raspberrypi3-debian-python:3.6.8-build 
INFO[0002] Built cross stage deps: map[]                
INFO[0002] Retrieving image manifest balenalib/raspberrypi3-debian-python:3.6.8-build 
INFO[0002] Retrieving image balenalib/raspberrypi3-debian-python:3.6.8-build 
INFO[0003] Retrieving image manifest balenalib/raspberrypi3-debian-python:3.6.8-build 
INFO[0003] Retrieving image balenalib/raspberrypi3-debian-python:3.6.8-build 
INFO[0004] Executing 0 build triggers                   
INFO[0004] Unpacking rootfs as cmd RUN [ "cross-build-start" ] requires it. 
INFO[0038] RUN [ "cross-build-start" ]                  
INFO[0038] Taking snapshot of full filesystem...        
INFO[0049] cmd: /usr/bin/cross-build-start              
INFO[0049] args: []                                     
INFO[0049] Running: [/usr/bin/cross-build-start]        
2021/01/26 05:44:09 link /bin/sh.real /bin/sh: no such file or directory
error building image: error building stage: failed to execute command: waiting for process to exit: exit status 1

The behaviour of the Balena image is explained here https://www.balena.io/blog/building-arm-containers-on-any-x86-machine-even-dockerhub/#improvingthesyntax

Additional Information

• Dockerfile

  
  FROM balenalib/raspberrypi3-debian-python:3.6.8-build

RUN [ "cross-build-start" ]

Some PiP stuff, but we don't even get here

RUN [ "cross-build-end" ]

ENTRYPOINT ["python", "application", "--production"]

 - Build Context
not needed
 - Kaniko Image (fully qualified with digest)
 `gcr.io/kaniko-project/executor:debug`

 **Triage Notes for the Maintainers**
 <!-- 🎉🎉🎉 Thank you for an opening an issue !!! 🎉🎉🎉
We are doing our best to get to this. Please help us by helping us prioritize your issue by filling the section below -->

 | **Description** | **Yes/No** |
 |----------------|---------------|
 | Please check if this a new feature you are proposing        | <ul><li>- [ ] </li></ul>|
 | Please check if the build works in docker but not in kaniko | <ul><li>- [x] </li></ul>| 
 | Please check if this error is seen when you use `--cache` flag | <ul><li>- [ ] </li></ul>|
 | Please check if your dockerfile is a multistage dockerfile | <ul><li>- [ ] </li></ul>| 

[tekn0ir]

The ticket #1292 seems to be closed prematurely and should be opened again, right @tejal29? Please take a look at the new posts in that ticket and try to understand what you are missing!

It is really important to get the RUN "exec form" working, see https://docs.docker.com/engine/reference/builder/#run.

[bearprevent]

This might be a balena not a kaniko issue.

The problem is that cross-build-start takes for some reason the wrong execution path. /usr/bin/cross-build-start is a link to /usr/bin/resin-xbuildand runningcross-build-startshould firstly link/bin/shto/bin/sh.realand secondly link/usr/bin/resin-xbuildto/bin/sh`.

However, it wants to link the non existing path /bin/sh.real to /bin/sh: this should happen when resin-xbuild is used as a shell wrapper or when cross-build-end is called.

A workaround is to provide the missing file /bin/sh.real. An other workaround is to build an own resin-xbuild. I took the code from https://github.com/balena-io-library/armv7hf-debian-qemu/blob/master/resin-xbuild.go.

Workaround 1: Get the missing sh.real

Dockerfile

FROM balenalib/armv7hf-debian-python AS base

FROM balenalib/armv7hf-debian-python                                                                                                                                                                                                                                                                      

COPY --from=base /bin/sh /bin/sh.real

RUN ["cross-build-start"]

RUN pip install --upgrade pip

RUN ["cross-build-end"]

Test script

docker run --rm -it --volume ./:/test gcr.io/kaniko-project/executor:debug --context /test --dockerfile /test/Dockerfile --no-push

Workaround 2: Use own built resin-xbuild

Dockerfile

FROM balenalib/armv7hf-debian-python                                                                                                                                                                                                                                                                      

COPY resin-xbuild /usr/bin/resin-xbuild

RUN ["cross-build-start"]

RUN pip install --upgrade pip

RUN ["cross-build-end"]

Test script

wget https://github.com/balena-io-library/armv7hf-debian-qemu/blob/master/resin-xbuild.go
go build resin-xbuild.go
docker run --rm -it --volume ./:/test gcr.io/kaniko-project/executor:debug --context /test --dockerfile /test/Dockerfile --no-push