search

balena-os / balena-supervisor

Balena Supervisor: balena's agent on devices.
https://balena.io
Other
148 stars 61 forks source link

Updates locked on Factory Build with Supervisor 16.2.0 #2261

Closed jonmuell closed 3 months ago

jonmuell commented 3 months ago

We just tried to install some Generic x86_64 devices. After device startup, the containers are not started because updates are locked. See attached screenshot for more infos. This happens on both MBR and GPT versions. Activating the update lock override in the dashboard did not work either.

Bildschirmfoto 2024-04-05 um 09 04 34

The service pulseaudio-bridge does not even create a lockfile. The service logged in the console sometimes changes to a different one that does create one but only after the container has started, which it did not.

netliteIT commented 3 months ago

Same thing here ... immagine

pipex commented 3 months ago

Hi there, could you share the output of ls -laR /tmp/balena-supervisor/services/ from the HostOS terminal?

cywang117 commented 3 months ago

@jonmuell Did the error occur as soon as you upgraded to Supervisor 16.2.0, or when you pushed a release?

netliteIT commented 3 months ago

I've actually two identical Compulab IOT-gate-imx8 devices affected. One untouched after 16.2.0 upgrade one heavily touched.

Device A

Variant production. Just upgraded to 16.2.0 with no new sofware release pushed. Containers are up but logs are full of:

Taking update locks
Taking update locks
Updates are locked, retrying in 15m. Reason: Lockfile exists for { appId: 2012699, service: gear-driver-bizerba-listener }

ls -laR /tmp/balena-supervisor/services/

/tmp/balena-supervisor/services/:
total 0
drwxr-xr-x  3 root root  60 Mar 28 08:12 .
drwxr-xr-x  3 root root 100 Mar 28 08:12 ..
drwxr-xr-x 12 root root 240 Mar 28 16:54 2012699

/tmp/balena-supervisor/services/2012699:
total 0
drwxr-xr-x 12 root root 240 Mar 28 16:54 .
drwxr-xr-x  3 root root  60 Mar 28 08:12 ..
drwxrwxrwt  2 root root  40 Apr  6 06:39 dashboard
drwxrwxrwt  2 root root  40 Apr  6 06:39 etcd-browser
drwxrwxrwt  2 root root  40 Apr  6 06:39 frontend
drwxrwxrwt  2 root root  40 Apr  6 06:39 gear-consumer
drwxrwxrwt  2 root root  40 Apr  6 06:39 gear-driver
drwxrwxrwt  2 root root  40 Apr  6 06:39 gear-driver-bizerba-listener
drwxrwxrwt  2 root root  40 Apr  6 06:39 rproxy
drwxrwxrwt  2 root root  40 Apr  6 06:39 srv-etcd-1
drwxrwxrwt  2 root root  40 Apr  6 06:39 srv-mq-1
drwxrwxrwt  2 root root  40 Apr  6 06:39 srv-redis

/tmp/balena-supervisor/services/2012699/dashboard:
total 0
drwxrwxrwt  2 root root  40 Apr  6 06:39 .
drwxr-xr-x 12 root root 240 Mar 28 16:54 ..

/tmp/balena-supervisor/services/2012699/etcd-browser:
total 0
drwxrwxrwt  2 root root  40 Apr  6 06:39 .
drwxr-xr-x 12 root root 240 Mar 28 16:54 ..

/tmp/balena-supervisor/services/2012699/frontend:
total 0
drwxrwxrwt  2 root root  40 Apr  6 06:39 .
drwxr-xr-x 12 root root 240 Mar 28 16:54 ..

/tmp/balena-supervisor/services/2012699/gear-consumer:
total 0
drwxrwxrwt  2 root root  40 Apr  6 06:39 .
drwxr-xr-x 12 root root 240 Mar 28 16:54 ..

/tmp/balena-supervisor/services/2012699/gear-driver:
total 0
drwxrwxrwt  2 root root  40 Apr  6 06:39 .
drwxr-xr-x 12 root root 240 Mar 28 16:54 ..

/tmp/balena-supervisor/services/2012699/gear-driver-bizerba-listener:
total 0
drwxrwxrwt  2 root root  40 Apr  6 06:39 .
drwxr-xr-x 12 root root 240 Mar 28 16:54 ..

/tmp/balena-supervisor/services/2012699/rproxy:
total 0
drwxrwxrwt  2 root root  40 Apr  6 06:39 .
drwxr-xr-x 12 root root 240 Mar 28 16:54 ..

/tmp/balena-supervisor/services/2012699/srv-etcd-1:
total 0
drwxrwxrwt  2 root root  40 Apr  6 06:39 .
drwxr-xr-x 12 root root 240 Mar 28 16:54 ..

/tmp/balena-supervisor/services/2012699/srv-mq-1:
total 0
drwxrwxrwt  2 root root  40 Apr  6 06:39 .
drwxr-xr-x 12 root root 240 Mar 28 16:54 ..

/tmp/balena-supervisor/services/2012699/srv-redis:
total 0
drwxrwxrwt  2 root root  40 Apr  6 06:39 .
drwxr-xr-x 12 root root 240 Mar 28 16:54 ..

Device B

Variant development. Just upgraded to 16.2.0 like device A but tried everything. New release. Rollback. Clean. Manual remove of files in /tmp/. balena system prune. etc. Containers are in Downloaded status and logs are full of:

Taking update locks
Taking update locks
Updates are locked, retrying in 15m. Reason: Lockfile exists for { appId: 2012699, service: gear-driver-bizerba-listener }

ls -laR /tmp/balena-supervisor/services/

/tmp/balena-supervisor/services/:
total 0
drwxr-xr-x  3 root root  60 Apr  5 19:58 .
drwxr-xr-x  3 root root 100 Apr  5 19:59 ..
drwxr-xr-x 13 root root 260 Apr  5 20:00 2012699

/tmp/balena-supervisor/services/2012699:
total 0
drwxr-xr-x 13 root root 260 Apr  5 20:00 .
drwxr-xr-x  3 root root  60 Apr  5 19:58 ..
drwxrwxrwt  2 root root  40 Apr  6 06:55 dashboard
drwxrwxrwt  2 root root  40 Apr  6 06:55 etcd-browser
drwxrwxrwt  2 root root  40 Apr  6 06:55 frontend
drwxrwxrwt  2 root root  40 Apr  6 06:55 gear-consumer
drwxrwxrwt  2 root root  40 Apr  6 06:55 gear-driver
drwxrwxrwt  2 root root  40 Apr  6 06:55 gear-driver-bizerba-listener
drwxrwxrwt  2 root root  40 Apr  5 21:09 ntp
drwxrwxrwt  2 root root  40 Apr  6 06:55 rproxy
drwxrwxrwt  2 root root  40 Apr  6 06:55 srv-etcd-1
drwxrwxrwt  2 root root  40 Apr  6 06:55 srv-mq-1
drwxrwxrwt  2 root root  40 Apr  6 06:55 srv-redis

/tmp/balena-supervisor/services/2012699/dashboard:
total 0
drwxrwxrwt  2 root root  40 Apr  6 06:55 .
drwxr-xr-x 13 root root 260 Apr  5 20:00 ..

/tmp/balena-supervisor/services/2012699/etcd-browser:
total 0
drwxrwxrwt  2 root root  40 Apr  6 06:55 .
drwxr-xr-x 13 root root 260 Apr  5 20:00 ..

/tmp/balena-supervisor/services/2012699/frontend:
total 0
drwxrwxrwt  2 root root  40 Apr  6 06:55 .
drwxr-xr-x 13 root root 260 Apr  5 20:00 ..

/tmp/balena-supervisor/services/2012699/gear-consumer:
total 0
drwxrwxrwt  2 root root  40 Apr  6 06:55 .
drwxr-xr-x 13 root root 260 Apr  5 20:00 ..

/tmp/balena-supervisor/services/2012699/gear-driver:
total 0
drwxrwxrwt  2 root root  40 Apr  6 06:55 .
drwxr-xr-x 13 root root 260 Apr  5 20:00 ..

/tmp/balena-supervisor/services/2012699/gear-driver-bizerba-listener:
total 0
drwxrwxrwt  2 root root  40 Apr  6 06:55 .
drwxr-xr-x 13 root root 260 Apr  5 20:00 ..

/tmp/balena-supervisor/services/2012699/ntp:
total 0
drwxrwxrwt  2 root root  40 Apr  5 21:09 .
drwxr-xr-x 13 root root 260 Apr  5 20:00 ..

/tmp/balena-supervisor/services/2012699/rproxy:
total 0
drwxrwxrwt  2 root root  40 Apr  6 06:55 .
drwxr-xr-x 13 root root 260 Apr  5 20:00 ..

/tmp/balena-supervisor/services/2012699/srv-etcd-1:
total 0
drwxrwxrwt  2 root root  40 Apr  6 06:55 .
drwxr-xr-x 13 root root 260 Apr  5 20:00 ..

/tmp/balena-supervisor/services/2012699/srv-mq-1:
total 0
drwxrwxrwt  2 root root  40 Apr  6 06:55 .
drwxr-xr-x 13 root root 260 Apr  5 20:00 ..

/tmp/balena-supervisor/services/2012699/srv-redis:
total 0
drwxrwxrwt  2 root root  40 Apr  6 06:55 .
drwxr-xr-x 13 root root 260 Apr  5 20:00 ..
cywang117 commented 3 months ago

I released Supervisor 16.2.1 which should patch this. You can find details about the patch here: https://github.com/balena-os/balena-supervisor/pull/2263

Please let us know if the issue persists after upgrading the Supervisor.

netliteIT commented 3 months ago

It solves on untouched Device A. On Device B some containers doesn't update and start. But I think it's unrelated to bugfix. Thanks.

jonmuell commented 3 months ago

For us, all devices are working again on supervisor 16.2.2. Thanks for the quick fix!