Livepush / Supervisor loses track of changes, get stuck and keeps pointing to old image

StefKors commented 3 years ago

Problem:

When developing on my NUC I've noticed Livepush or the supervisor get stuck pointing at old livepush images. This happens sometimes and I haven't been able to reliably reproduce the issue.

Symptoms:

First seen about 10 months ago and seen it recently again. The symptoms are:

The site developing via livepush not updating state after saving a file.
Site shows status 503
Stopping the balena push process and running it again won't start watching for changes.

See gist for example: https://gist.github.com/StefKors/ec7cbda63dbd51cf51e40a3f670cfaaa

Current workarounds:

Running balena image prune

Resetting the device state (sometimes a restart is required):

curl -X POST --header "Content-Type:application/json" "http://jel.ly.fish.local:48484/v2/local/target-state"
   -d '{"local": {"name": "restless-frost","config": {},"apps": {}},"dependent": {"config": {}}}'

Related Conversations:

https://www.flowdock.com/app/rulemotion/resin-starters/threads/yuS9WCzSeUGm4e_FwBp2bbc7ALU

StefKors commented 3 years ago

@richbayliss you have also seen this issue, do you have any information or insight to add here?

LucianBuzzo commented 3 years ago

AFAICT, the NUC images come with v11 of the supervisor and are way behind the current balena-meta version. It's possible that this issue is resolved in the latest v12 supervisor version.

StefKors commented 3 years ago

Ran into this again on cli version: 12.37.0 https://gist.github.com/StefKors/0777f107ba0e10d9d59b4320f21a85d1

pdcastro commented 3 years ago

@20k-ultra may already know about this issue, but mentioning him just in case, as I know that he was recently working on some livepush issues where the supervisor may play a role. 🙏

balena-os / balena-supervisor