`balena build` does not specify it's using emulation when using the `--application` switch on non-native targets

[hedss]

• balena CLI version: 11.9.3
• Operating system version: macOS 10.14.5
• 32/64 bit OS and processor: 64-bit macOS on 64-bit Intel processor
• Install method: npm
• If npm install, Node.js and npm version: Node v10.16.0 / NPM 6.9.0

When building using the --application switch, balenaCLI uses the device type and architecture by querying the balena API. This allows it to correctly determine whether an emulated build if required, or a native one.

However, it does not specify that it's running out an emulated build if doing so, which differs to that when using --emulated. For example:

balena build --application cliApp
[Info]    Creating default composition with source: /Work/Support/MasterClasses/repos/balena-cli-masterclass
[Info]    Building for armv7hf/fincm3
[Build]   Built 1 service in 1:31
[Build]   main Image size: 213.80 MB
[Success] Build succeeded!

differs to :

balena build --application cliApp --emulated
[Info]    Creating default composition with source: /Work/Support/MasterClasses/repos/balena-cli-masterclass
[Info]    Building for armv7hf/fincm3
[Info]    Emulation is enabled
[Build]   Built 1 service in 1:05
[Build]   main Image size: 219.15 MB
[Success] Build succeeded!

Whilst during the build steps, if observed, a user can note that QEMU is being used, these build steps are cleared once the build has finished, so if being used from, for example, a CI system, there's no evidence that an emulated build occurred, which can cause consternation and confusion that a valid target has not been produced.

We should ensure that if an emulated build is occurring, regardless of how that information was derived, it is clearly marked as such.

[pdcastro]

I've created PR #1416 but there is a twist to this issue. Even before the PR, the CLI would already treat Docker for Mac as a special case, because it "provides binfmt_misc multi-architecture support" and uses emulation as needed, regardless of whether the --emulated flag is provided and without CLI intervention / support. Yet I was not expecting to see [Info] Emulation is enabled in one of the build output excerpts above, because it should only be printed when the CLI itself sets up QEMU (downloads and copies it to the built image), which it does not do for Docker for Mac. On investigation, I found that older Docker for Mac engines/daemons (like 18.06) used the string "Docker for Mac" as the value of the Operating System field returned by the info() API call, whereas newer versions (like 19.03) use the value "Docker Desktop", and the CLI was not correctly detecting the newer versions. The PR I have at the moment addresses this detection issue.

To date, the CLI does not attempt to determine whether the Docker engine/daemon (e.g. Docker for Mac) will actually make use of emulation, when Docker is capable of. It sounds like it may be something dynamic at binary execution time, for each binary (not sure). What's easier to do, and I've done in the PR (not sure if it is good enough), is to print additional INFO lines that Docker Desktop was detected and Docker itself will determine and enable emulation as needed, without CLI intervention and regardless of the --emulated option. The PR is there for review, please share any thoughts...

[hedss]

@pdcastro So the issue is generally that there is some intelligence going on here, be it either in Docker for Mac or otherwise. I think the underlying issue is that we should be doing some intelligence here if we detect we can build it for a certain architecture automatically. Let's look at what happens under Linux:

heds@media-pc:~/Work/balena-cli-masterclass$ balena build --application cliApp
[Info]    Creating default composition with source: /home/heds/Work/balena-cli-masterclass
[Info]    Building for armv7hf/fincm3
[Build]   Built 1 service in 0:17
[Build]   main [=============>       ]  66% Step 4/6: standard_init_linux.go:190: exec user process caused "exec format error"
[Error]   Build failed

Now, whilst the issue is that it's Docker on different platforms causing the issue, we should know what's happening and determine if we can build it. Maybe this should be a different issue now, thinking about it. But ultimately, there's a problem with consistency across platforms and builds.

[pdcastro]

Note from the CLI check-in of 9 September: we may not be able to detect whether users are making use of binfmt_misc, so the idea is to parse the build output for key error messages such as "exec format error" and then print a warning to the user that they could try repeating the build with the --emulated flag.

[jellyfish-bot]

[rahul-thakoor] This issue has attached support thread https://jel.ly.fish/b3624426-5f5e-4491-a9af-fc4dd141aaf6