Continuous updates

[splitice]

Description

Failure to start with repeating error message:

Jan 31 00:42:35 d1d38bd resin-supervisor[18545]: [error]   Scheduling another update attempt in 16000ms due to failure:  Error: (HTTP code 500) server error - OCI runtime create failed: container with id exists: 7ddbcbe5f6fb1a8e314936ca0fed8afdee03ec68fedd71c05d638506f118beb3: unknown

Steps to reproduce the issue:

1. Unknown: assumed corruption.

Describe the results you received:

Constant looping of attempting to update the container (no update available, target release = current release). Container exists.

Attempted to fix via:

balena container rm 7ddbcbe5f6fb1a8e314936ca0fed8afdee03ec68fedd71c05d638506f118beb3

Results in:

Jan 31 00:47:11 d1d38bd fce6d5e0e686[13644]: [event]   Event: Service install {"service":{"appId":1158441,"serviceId":64983,"serviceName":"main","releaseId":1154562}}
Jan 31 00:47:11 d1d38bd resin-supervisor[18545]: [event]   Event: Service install {"service":{"appId":1158441,"serviceId":64983,"serviceName":"main","releaseId":1154562}}
Jan 31 00:47:12 d1d38bd fce6d5e0e686[13644]: [event]   Event: Service installed {"service":{"appId":1158441,"serviceId":64983,"serviceName":"main","releaseId":1154562}}
Jan 31 00:47:12 d1d38bd resin-supervisor[18545]: [event]   Event: Service installed {"service":{"appId":1158441,"serviceId":64983,"serviceName":"main","releaseId":1154562}}
Jan 31 00:47:12 d1d38bd fce6d5e0e686[13644]: [event]   Event: Service start {"service":{"appId":1158441,"serviceId":64983,"serviceName":"main","releaseId":1154562}}
Jan 31 00:47:12 d1d38bd resin-supervisor[18545]: [event]   Event: Service start {"service":{"appId":1158441,"serviceId":64983,"serviceName":"main","releaseId":1154562}}

And repaired.

Output of balena-engine version:

Client:
 Version:           18.09.10-dev
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        7cb464a406748016f2df0c31a9851d20456a3d31
 Built:             Fri Nov  1 05:04:49 2019
 OS/Arch:           linux/arm
 Experimental:      false

Server:
 Engine:
  Version:          18.09.10-dev
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       7cb464a406748016f2df0c31a9851d20456a3d31
  Built:            Fri Nov  1 05:04:49 2019
  OS/Arch:          linux/arm
  Experimental:     true

Output of balena-engine info:

Containers: 2
 Running: 2
 Paused: 0
 Stopped: 0
Images: 3
Server Version: 18.09.10-dev
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge host ipvlan null
 Log: journald json-file local
Swarm: 
 NodeID: 
 Is Manager: false
 Node Address: 
Runtimes: bare runc
Default Runtime: runc
Init Binary: balena-engine-init
containerd version: 
runc version: b3d5036b6520060db00c81352d533a5dd919aaaf
init version: 949e6fa-dirty (expected: fec3683b971d9)
Kernel Version: 4.19.84
Operating System: balenaOS 2.45.0+rev1
OSType: linux
Architecture: armv7l
CPUs: 4
Total Memory: 498.2MiB
Name: d1d38bd
ID: B4UI:ZMTZ:DBLW:YRGY:NMVO:4PJV:NV6Q:PJIZ:R7DR:7MYA:77DY:BPM3
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface

[splitice]

Tag @brdrcol

Perhaps this may be better suited as a bug against resin-supervisor.

[robertgzr]

@splitice did you experience this frequently? I agree this is probably more of a supervisor issue and I would attempt to troubleshoot this by restarting supervisor..

If this is a problem for you, please do reopen this ticket