Browser unable to start

[TechplexEngineer]

It seems the latest arm v7 image published does not run on the latest balena image for raspberry pi 4

12.10.21 15:23:21 (-0400) Restarting service 'browser sha256:fa7ca3bc2ac4314d5f68595bb8cb219c1b9259cffa39ae6f4617f1024543104c'
12.10.21 15:23:21 (-0400)  browser  Running balena base image entrypoint...
12.10.21 15:23:22 (-0400)  browser  setting xserver-xorg-legacy/xwrapper/allowed_users from configuration file
12.10.21 15:23:22 (-0400)  browser  Using default chromium flags
12.10.21 15:23:22 (-0400)  browser  Enabling GPU acceleration
12.10.21 15:23:22 (-0400)  browser  Using fullscreen: 1280,720
12.10.21 15:23:22 (-0400)  browser  Using default window position
12.10.21 15:23:22 (-0400)  browser  Enabling kiosk mode
12.10.21 15:23:22 (-0400)  browser  Enabling cursor
12.10.21 15:23:27 (-0400) Service exited 'browser sha256:fa7ca3bc2ac4314d5f68595bb8cb219c1b9259cffa39ae6f4617f1024543104c'
12.10.21 15:24:28 (-0400) Restarting service 'browser sha256:fa7ca3bc2ac4314d5f68595bb8cb219c1b9259c

when using the pre-published image named balenablocks/browser

If I build the browser container image as a part of my build the browser starts successfully.

If I change:

services:

  browser:
    image: balenablocks/browser

to:

services:

  browser:
    build:
      context: "./balena"

The build takes significantly longer but the browser runs. it also seems that the log messages generated by this container are different (see below) that the messages of the published container (listed above).

12.10.21 15:25:12 (-0400) Started service 'browser sha256:c802e8cc77d9827687c9c071398bf6e6233c0837050e9869e51a694b8e5f5953'
12.10.21 15:25:12 (-0400)  browser  user.max_user_namespaces = 10000
12.10.21 15:25:13 (-0400)  browser  umount: /dev: target is busy.
12.10.21 15:25:13 (-0400)  browser  mount: /sys/kernel/debug: nodev already mounted on /sys/kernel/debug.
12.10.21 15:25:13 (-0400)  browser  Running balena base image entrypoint...
12.10.21 15:25:15 (-0400)  browser  setting xserver-xorg-legacy/xwrapper/allowed_users from configuration file
12.10.21 15:25:15 (-0400)  browser  balenaBlocks browser version: 2.0.5
12.10.21 15:25:15 (-0400)  browser  Setting CPU Scaling Governor to 'performance'
12.10.21 15:25:15 (-0400)  browser  Enabling cursor
12.10.21 15:25:15 (-0400)  browser  chown: cannot access '/data': No such file or directory
12.10.21 15:25:15 (-0400)  browser  hostname: No address associated with hostname
12.10.21 15:25:15 (-0400)  browser  xauth:  file /home/chromium/.Xauthority does not exist
12.10.21 15:25:15 (-0400)  browser  xauth: (stdin):1:  bad display name "d145c77:0" in "add" command
12.10.21 15:25:15 (-0400)  browser  
12.10.21 15:25:15 (-0400)  browser  
12.10.21 15:25:15 (-0400)  browser  X.Org X Server 1.20.4
<SNIP>

This leads me to believe the balenablocks/browser image is out of date and needs to be republished on dockerhub.

[phil-d-wilson]

Thank for the help here. I found a bug with our build script. It was building the images OK, but failing to add them to the latest manifest. I've re-run the build and everything should be working when you pull the image, rather than build from source.

[edorgeville]

@phil-d-wilson Just deployed a new build of our app through balenaCloud and we're having the same issue. Looks like we're running on 2.0.5. Would really like the dockerhub tags to be kept instead of overriding latest as mentioned in #84. Is there a way to force the balenaCloud builder to pull the latest version of the published image?

root@afca01a:~# balena run --rm -it daa73fd482de cat VERSION
Unable to start udev, container must be run in privileged mode to start udev!
2.0.5
root@afca01a:~#

[edorgeville]

Actually, even in local mode, it seems like pulling latest still give 2.0.5:

root@afca01a:~# balena pull balenablocks/browser:latest
latest: Pulling from balenablocks/browser
Digest: sha256:e1eefff135a7122530b73b42249f8d412f9b0a3ecd4c6ebcd0f77602db7522c7
Status: Image is up to date for balenablocks/browser:latest
docker.io/balenablocks/browser:latest
root@afca01a:~# balena run --rm -it balenablocks/browser:latest cat VERSION
Unable to start udev, container must be run in privileged mode to start udev!
2.0.5
root@afca01a:~# 

[phil-d-wilson]

@edorgeville hey!

Actually V2.0.5 is the latest version, so you're up to date.

As for docker hub tags, we did talk about having version tags but decided that's not what we want to do. We want blocks to provide a function, and the image on docker hub to be the latest version of that function independent from how it's implemented. In theory we could change the underlying executable a block uses (in this case the browser provided) but as long as the block still provides the function (here: display a web page) we would still consider it the same block.

What you could do, if you really want to pin the version, is choose a commit level of the source repo and build/push your own image. The build script is in the source, and takes a docker hun organisation as a command line ARG.

Hope this helps, Phil

[edorgeville]

I fail to see why publishing tags would go against the concept you explained. Publishing tags wouldn't stop users from using latest, but would simply allow users that need to lock versions to do so. There are multiple valid reasons for this: quality assurance, bandwidth management (in case of remote fleets), security compliance, etc.

In our case, we cannot allow for service updates to be deployed on the field without us greenlighting it first, which is the current behavior. Today we had to rollback to our previous balena release because the latest tag was broken.

If the decision to not publish tags is final, we will transition away from using the pre-built docker image and build from source as you suggested; I'd have prefer we didn't, as it will add build time when developing in local mode, as well as add complexity to our monorepo.

Thanks

[TechplexEngineer]

I would love to be able to use tags as well.

[edorgeville]

I think I found a way to lock the image version even if tags are changed. First, pull the latest tag on a device running balenaOS:

$ balena pull balenablocks/browser:latest
latest: Pulling from balenablocks/browser
Digest: sha256:2e1244a6b55a0c1f265d0e4a83943044870559d2198c048cdd65987dce6d560d
Status: Image is up to date for balenablocks/browser:latest
docker.io/balenablocks/browser:latest

Note down the digest, in our case sha256:2e1244a6b55a0c1f265d0e4a83943044870559d2198c048cdd65987dce6d560d. Then, in your docker-compose.yml / Dockerfile, you can use balenablocks/browser@sha256:2e[...]0d, like so:

# docker-compose.yml
version: "2.0"
services:
 browser:
    image: "balenablocks/browser@sha256:2e1244a6b55a0c1f265d0e4a83943044870559d2198c048cdd65987dce6d560d"
    privileged: true # required for UDEV to find plugged in peripherals such as a USB mouse

# Dockerfile
FROM balenablocks/browser@sha256:2e1244a6b55a0c1f265d0e4a83943044870559d2198c048cdd65987dce6d560d

I tested it succesfully against a development device, and everything went well. We will test with our staging devices, then release the change. Knowing that there will be no unwanted updates to the browser block in our future releases will help me sleep at night. 😌

---

I was wondering if the older images were still present on Dockerhub, and it seems that yes, they are. I have a development device that went through multiple versions of the browser. Tags are missing, but the image and their digests are still on the device:

root@893c9e8:~# balena images balenablocks/browser --digests
REPOSITORY             TAG                 DIGEST                                                                    IMAGE ID            CREATED             SIZE
balenablocks/browser   latest              sha256:2e1244a6b55a0c1f265d0e4a83943044870559d2198c048cdd65987dce6d560d   9946c27073b7        8 days ago          968MB
balenablocks/browser   <none>              sha256:e1eefff135a7122530b73b42249f8d412f9b0a3ecd4c6ebcd0f77602db7522c7   98330da2bbf5        3 weeks ago         967MB
balenablocks/browser   <none>              sha256:c570130bcf3a0c4fb938b866cba56aac6a9db6af02478775223f46a962de22e3   fa7ca3bc2ac4        7 months ago        814MB

On my own workstation, I could still pull the image from 7 months ago:

$ docker pull balenablocks/browser@sha256:c570130bcf3a0c4fb938b866cba56aac6a9db6af02478775223f46a962de22e3
a330b6cecb98: Already exists
2aa5aad623cc: Already exists
f872d0ce380d: Already exists
5027522f8283: Already exists
2dfccb56e2c1: Already exists
fff51f9dadcb: Already exists
811c84be7362: Already exists
fee3f8e89f74: Already exists
d09b3de081de: Already exists
1f13cbc8f08c: Already exists
bdc50adb590c: Already exists
8df8b1311cbb: Already exists
0941804b2776: Already exists
cc3f5ee8b6e0: Already exists
5f08b5da940e: Already exists
f8de5256cb61: Already exists
ea72d23a1876: Already exists
5a787aca1ee1: Pull complete
dcfe57d63024: Pull complete
de6d133ca389: Pull complete
1ac35cbbe1d9: Pull complete
8ba90ef06dac: Pull complete
63bdbd650007: Pull complete
dfd3d87ccc45: Pull complete
01f77feb0d32: Pull complete
a221183f91f2: Pull complete
b6cf9d9b37a4: Pull complete
061344ec54a2: Pull complete
b474d75755fc: Pull complete
74aa2cddf6b2: Pull complete
fa5dd3b8d8b0: Pull complete
756dd405f927: Pull complete
Digest: sha256:e1eefff135a7122530b73b42249f8d412f9b0a3ecd4c6ebcd0f77602db7522c7
Status: Downloaded newer image for balenablocks/browser@sha256:e1eefff135a7122530b73b42249f8d412f9b0a3ecd4c6ebcd0f77602db7522c7
docker.io/balenablocks/browser@sha256:e1eefff135a7122530b73b42249f8d412f9b0a3ecd4c6ebcd0f77602db7522c7