search

balena-os / balena-supervisor

Balena Supervisor: balena's agent on devices.
https://balena.io
Other
148 stars 63 forks source link

Kill-then-download strategy isn't killing the containers with BALENA_SUPERVISOR_UPDATE_STRATEGY #1939

Closed vipulgupta2048 closed 2 years ago

vipulgupta2048 commented 2 years ago

Customer reported on a RPi zero 2 the call to kill the service comes after the image is downloaded when using BALENA_SUPERVISOR_UPDATE_STRATEGY=kill-then-download but when the RESIN_SUPERVISOR_UPDATE_STRATEGY=kill-then-download is used then everything works as expected.

Steps to reproduce:

  1. Set app's BALENA_SUPERVISOR_UPDATE_STRATEGY to delete-then-download
  2. Pinning the device to a different release
  3. Observed the logs below

The first two log lines are output from the running container. The line at (14:54:04) shows that the next release downloaded started. The call to kill the service isn't logged until line five (14:56:34), after the image has been downloaded (14:56:33).

27.04.22 14:53:50 (-0400)  api  ======== Running on http://0.0.0.0:80 ========
27.04.22 14:53:50 (-0400)  api  (Press CTRL+C to quit)
27.04.22 14:54:04 (-0400) Downloading delta for image 'registry2.balena-cloud.com/v2/d42ffa7e55694652a7e02eb37ab50a82@sha256:bf0124ff6d15aa698a6a4a3ded5f690b2fcbf44cc1e4151995f97aa591ff8a26'
27.04.22 14:56:33 (-0400) Downloaded image 'registry2.balena-cloud.com/v2/d42ffa7e55694652a7e02eb37ab50a82@sha256:bf0124ff6d15aa698a6a4a3ded5f690b2fcbf44cc1e4151995f97aa591ff8a26'
27.04.22 14:56:34 (-0400) Killing service 'api sha256:f739d5f084b14b4e4a022ee1bcc46832b8be7350f25d03d8fc8fa46514927010'
27.04.22 14:56:46 (-0400) Service exited 'api sha256:f739d5f084b14b4e4a022ee1bcc46832b8be7350f25d03d8fc8fa46514927010'
27.04.22 14:56:47 (-0400) Killed service 'api sha256:f739d5f084b14b4e4a022ee1bcc46832b8be7350f25d03d8fc8fa46514927010'
27.04.22 14:56:48 (-0400) Deleting image 'registry2.balena-cloud.com/v2/c8544ad2ec434346a4a8e31d69e213f9@sha256:412ffbd48828a95b21d7a6a65e7469c0bbdcd721e75740beb9eda6876d16c489'
27.04.22 14:57:04 (-0400) Deleted image 'registry2.balena-cloud.com/v2/c8544ad2ec434346a4a8e31d69e213f9@sha256:412ffbd48828a95b21d7a6a65e7469c0bbdcd721e75740beb9eda6876d16c489'
27.04.22 14:57:05 (-0400) Installing service 'api sha256:5d8bfcb99a1e2d655c6c7da2bebecf5aab4a3795cc5b610068e6e465694d2307'
27.04.22 14:57:07 (-0400) Installed service 'api sha256:5d8bfcb99a1e2d655c6c7da2bebecf5aab4a3795cc5b610068e6e465694d2307'
27.04.22 14:57:07 (-0400) Starting service 'api sha256:5d8bfcb99a1e2d655c6c7da2bebecf5aab4a3795cc5b610068e6e465694d2307'
27.04.22 14:57:09 (-0400) Started service 'api sha256:5d8bfcb99a1e2d655c6c7da2bebecf5aab4a3795cc5b610068e6e465694d2307'
  1. Set `RESIN_SUPERVISOR_UPDATE_STRATEGY=kill-then-download
  2. Repin the release
  3. Logs the customer observed now

Killing first then downloading as expected.

27.04.22 16:08:28 (-0400)  api  ======== Running on http://0.0.0.0:80 ========
27.04.22 16:08:28 (-0400)  api  (Press CTRL+C to quit)
27.04.22 16:08:39 (-0400) Killing service 'api sha256:d07224f598b3340b4d8aff35a5cf703fd4cc2b62eccc4747f582664ace3cd802'
27.04.22 16:08:51 (-0400) Service exited 'api sha256:d07224f598b3340b4d8aff35a5cf703fd4cc2b62eccc4747f582664ace3cd802'
27.04.22 16:08:51 (-0400) Killed service 'api sha256:d07224f598b3340b4d8aff35a5cf703fd4cc2b62eccc4747f582664ace3cd802'
27.04.22 16:08:52 (-0400) Downloading delta for image 'registry2.balena-cloud.com/v2/d42ffa7e55694652a7e02eb37ab50a82@sha256:bf0124ff6d15aa698a6a4a3ded5f690b2fcbf44cc1e4151995f97aa591ff8a26'
27.04.22 16:10:48 (-0400) Downloaded image 'registry2.balena-cloud.com/v2/d42ffa7e55694652a7e02eb37ab50a82@sha256:bf0124ff6d15aa698a6a4a3ded5f690b2fcbf44cc1e4151995f97aa591ff8a26'
27.04.22 16:10:50 (-0400) Deleting image 'registry2.balena-cloud.com/v2/6124c6e4e354be2d11ea065ee8123bb6@sha256:5b4489134e8b241cf2653e739b8cac7120f713ea7d274fd47b46a3c801f070df'
27.04.22 16:10:56 (-0400) Deleted image 'registry2.balena-cloud.com/v2/6124c6e4e354be2d11ea065ee8123bb6@sha256:5b4489134e8b241cf2653e739b8cac7120f713ea7d274fd47b46a3c801f070df'
27.04.22 16:10:57 (-0400) Installing service 'api sha256:5d8bfcb99a1e2d655c6c7da2bebecf5aab4a3795cc5b610068e6e465694d2307'
27.04.22 16:10:58 (-0400) Installed service 'api sha256:5d8bfcb99a1e2d655c6c7da2bebecf5aab4a3795cc5b610068e6e465694d2307'
27.04.22 16:10:58 (-0400) Starting service 'api sha256:5d8bfcb99a1e2d655c6c7da2bebecf5aab4a3795cc5b610068e6e465694d2307'
27.04.22 16:11:00 (-0400) Started service 'api sha256:5d8bfcb99a1e2d655c6c7da2bebecf5aab4a3795cc5b610068e6e465694d2307'

Possiblities

jellyfish-bot commented 2 years ago

[vipulgupta2048] This issue has attached support thread https://jel.ly.fish/4dbd2b6e-0f60-42e5-a11c-cb85c2873984

jack-solomomo commented 2 years ago

I originally reported this on Balena Forums. This was tested across multiple devices that are all Raspberry Pi Zero 2 W running BalenaOS 2.94.4 development and supervisor 13.0.0. Let me know if there is any other information that would be helpful. Thanks

pipex commented 2 years ago

Hi @jack-solomomo, thanks for reporting this. Unfortunately you ran into some missing documentation. The recommended way to set the update strategy is through a composition label io.balena.features.update-strategy. Setting an env var is a mechanism made for backwards compatibility with single container supervisors but is probably going to be deprecated in the future. We have updated the documentation to make it clearer (see https://github.com/balena-io/docs/pull/2265) and it should make it to the documentation page soon.

pipex commented 2 years ago

For extra context. The config vars are handled by the API, that converts them to labels. Here is a related issue, although the consensus is that using the labels is the correct mechanism

https://github.com/balena-io/open-balena-api/issues/735

jack-solomomo commented 2 years ago

@pipex Thanks! That makes a lot of sense. Per-container update strategies look really useful.