"stat /tmp/qemu-execve: no such file or directory" when doing emulated build on Linux

[saintaardvark]

Hi there -- I'm running into problems when trying to do an emulated build, following the instructions in the CLI Masterclass.

Here's the command and error I'm seeing:

$ DEBUG=1 balena build --application cliApp --emulated
debug] original argv0="balena" argv=[/home/hugh/.dotfiles/bin/balena-cli/balena,/snapshot/versioned-source/bin/balena,build,--application,cliApp,--emulated] length=6
[Debug]   Parsing input...
[Debug]   Loading project...
[Debug]   Resolving project...
[Info]    No "docker-compose.yml" file found at "/home/hugh/dev/github.com/balena-io/balena-cli-masterclass"
[Info]    Creating default composition with source: "/home/hugh/dev/github.com/balena-io/balena-cli-masterclass"
[Debug]   Creating project...
[Info]    Building for aarch64/raspberrypi4-64
[Info]    Emulation is enabled
[Debug]   Found build tasks:.. /
[Debug]       main: build [.]
[Debug]   Resolving services with [raspberrypi4-64|aarch64]
[Debug]   Found project types: -
[Debug]       main: Dockerfile.template
[Debug]   Prepared tasks; building...
[Build]   Built 1 service in 0 seconds
[Build]   main [===============>     ]  75% Step 6/8:  ---> a5067b09f7b3
[Error]   Build failed
OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"/tmp/qemu-execve\": stat /tmp/qemu-execve: no such file or directory": unknown
Error: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"/tmp/qemu-execve\": stat /tmp/qemu-execve: no such file or directory": unknown
    at Stream.<anonymous> (/snapshot/versioned-source/node_modules/resin-docker-build/build/builder.js:181:23)
    at Stream.stream.write (/snapshot/versioned-source/node_modules/through/index.js:26:11)

(I've left out the rest of the backtrace; let me know if you'd like it.)

The details are:

• Balena CLI version: 11.31.7
• OS: Linux, x86-64, Fedora 31; I've duplicated this with SELinux both on and off
• The app device type: raspberrypi-64

This looks sort of similar to #22, but seems different enough to warrant a separate issue. If there's a better place to file this, or if you need any further info, please let me know.

[pdcastro]

@saintaardvark, I have not been able to reproduce this error. Something to check: the CLI downloads relevant versions of the QEMU emulator at runtime (as needed depending on the device type) to the following folders, which should have filesystem permissions to the current user -- check e.g. root vs regular user:

$ ls -la ~/.balena/bin/
-rwxr-xr-x 1 osboxes osboxes 6192520 Apr 30 17:01 qemu-execve-aarch64-v4.0.0-balena
-rwxr-xr-x 1 osboxes osboxes 6192520 Apr 30 16:56 qemu-execve-aarch64-v4.0.0+balena2
-rwxr-x--- 1 osboxes osboxes 5606984 Apr 29 18:49 qemu-execve-armv7hf-v4.0.0-balena
-rwxr-x--- 1 osboxes osboxes 5606984 Apr 29 18:49 qemu-execve-armv7hf-v4.0.0+balena1
-rwxr-xr-x 1 osboxes osboxes 5606984 Apr 29 18:27 qemu-execve-armv7hf-v4.0.0+balena2

osboxes@osboxes:~/balena/balena-io/balena-cli-masterclass$ ls -la .balena
drwxr-xr-x 2 osboxes osboxes    4096 Apr 30 17:47 .
drwxr-xr-x 6 osboxes osboxes    4096 Apr 30 17:46 ..
-rwxr-xr-x 1 osboxes osboxes 6192520 Apr 30 17:49 qemu-execve

Here's what I get on Ubuntu 18.04.3 with the CLI version you reported, 11.31.7:

osboxes@osboxes:~/Downloads$ curl -LO https://github.com/balena-io/balena-cli/releases/download/v11.31.7/balena-cli-v11.31.7-linux-x64-standalone.zip

osboxes@osboxes:~/Downloads$ unzip balena-cli-v11.31.7-linux-x64-standalone.zip

osboxes@osboxes:~/Downloads$ mv balena-cli balena-cli-11.31.7

osboxes@osboxes:~/balena/balena-io$ git clone https://github.com/balena-io/balena-cli-masterclass.git

osboxes@osboxes:~/balena/balena-io$ cd balena-cli-masterclass/

osboxes@osboxes:~/balena/balena-io/balena-cli-masterclass$ sed -i 's/node:8/node:8-20200318/' Dockerfile.template

osboxes@osboxes:~/balena/balena-io/balena-cli-masterclass$ git diff
-FROM balenalib/%%BALENA_MACHINE_NAME%%-node:8
+FROM balenalib/%%BALENA_MACHINE_NAME%%-node:8-20200318

osboxes@osboxes:~/balena/balena-io/balena-cli-masterclass$ DEBUG=1 ~/Downloads/balena-cli-11.31.7/balena build --application test-rpi4 --emulated --logs --nocache 
[debug] original argv0="/home/osboxes/Downloads/balena-cli-11.31.7/balena" argv=[/home/osboxes/Downloads/balena-cli-11.31.7/balena,/snapshot/versioned-source/bin/balena,build,--application,test-rpi4,--emulated,--logs,--nocache] length=8
[Debug]   Parsing input...
[Debug]   Loading project...
[Debug]   Resolving project...
[Info]    No "docker-compose.yml" file found at "/home/osboxes/balena/balena-io/balena-cli-masterclass"
[Info]    Creating default composition with source: "/home/osboxes/balena/balena-io/balena-cli-masterclass"
[Debug]   Creating project...
[Info]    Building for aarch64/raspberrypi4-64
[Build]   Building services...
[Build]   main Preparing...
[Info]    Emulation is enabled
[Debug]   Found build tasks:
[Debug]       main: build [.]
[Debug]   Resolving services with [raspberrypi4-64|aarch64]
[Debug]   Found project types:
[Debug]       main: Dockerfile.template
[Debug]   Prepared tasks; building...
[Build]   main Step 1/7 : FROM balenalib/raspberrypi4-64-node:8-20200318
[Build]   main  ---> 8f6b562c62c6
[Build]   main Step 2/7 : COPY [".balena/qemu-execve","/tmp/qemu-execve"]
[Build]   main  ---> 47c6cd0c9e65
[Build]   main Step 3/7 : WORKDIR /usr/src/app
[Build]   main  ---> Running in 92803de1339a
[Build]   main Removing intermediate container 92803de1339a
[Build]   main  ---> 9ebf72237889
[Build]   main Step 4/7 : COPY ["package.json","package-lock.json","./"]
[Build]   main  ---> b61717e68eaa
[Build]   main Step 5/7 : RUN ["/tmp/qemu-execve","-execve","/bin/sh","-c","npm install --ci --production     && npm cache clean --force     && rm -rf /tmp/*"]
[Build]   main  ---> Running in cc3f72fef931
[Build]   main up to date in 2.131s
[Build]   main found 0 vulnerabilities
[Build]   main npm WARN using --force I sure hope you know what you are doing.
[Build]   main Removing intermediate container cc3f72fef931
[Build]   main  ---> 93f3627c5fbc
[Build]   main Step 6/7 : COPY ["src/","./src/"]
[Build]   main  ---> b9ef13ea435d
[Build]   main Step 7/7 : CMD ["npm","start"]
[Build]   main  ---> Running in f62081fa8a3d
[Build]   main Removing intermediate container f62081fa8a3d
[Build]   main  ---> 16eb72c8c005
[Build]   main Successfully built 16eb72c8c005
[Build]   main Successfully tagged balena-cli-masterclass_main:latest
[Build]   main Waiting...
[Build]   Built 1 service in 0:19
[Success] Build succeeded!

Note the sed and git diff commands there. Without those, I get a QEMU error but it is not the error you have reported. It is the following different issue: https://github.com/balena-io/balena-cli/issues/1788

The sed command can be avoided (using Dockerfile.template unmodified) by using CLI v11.31.26 or later. I have just tested with CLI v11.32.4.

Try reproducing the steps above, including re-cloning the masterclass repo and using a CLI zip package. (If the zip package works but the npm installation doesn't, try deleting the npm installation and re-installing.)

[saintaardvark]

Hi @pdcastro -- thanks very kindly for looking into this. On further investigation, I'm seeing some inconsistent behaviour on my machine:

• An emulated build, with RPi4 as the target, using zipped version of 11.31.7, now works with a freshly cloned copy of the masterclass.
• An emulated build, with Balena Fin as the target, using zipped version of 11.31.7, fails with the error I reported originally, with a freshly cloned copy of the masterclass.
• An emulated build, with either RPi4 or Balena Fin as the target, using zipped version of 11.32.3, now works with a freshly cloned copy of the masterclass.

The overall result is that a fresh clone of the repo works just fine with the latest version of the CLI. I'm not sure what's going on here; I'm happy to dig into this more, but I'm not sure it will be worth your time. For now, I'm willing to put this down as operator error. :grin: