docker-archive / classicswarm

Swarm Classic: a container clustering system. Not to be confused with Docker Swarm which is at https://github.com/docker/swarmkit
Apache License 2.0
5.75k stars 1.08k forks source link

docker swarm service not always updating #2896

Closed oppianmatt closed 4 years ago

oppianmatt commented 6 years ago

I have a service that doesn't always update. Docker service inspect will sometimes list is as updated or will sometimes just say updating for hours, like so:


ID:             y2aajwbddimk8qpds91jzduxs
Name:           pitchup_gunicorn
Labels:
 com.docker.stack.image=registry.gitlab.pitchup.com/pitchup/pitchup.com/master:767a4cc0-13456
 com.docker.stack.namespace=pitchup
Service Mode:   Replicated
 Replicas:      4
UpdateStatus:
 State:         updating
 Started:       18 hours ago
 Message:       update in progress
Placement:
 Constraints:   [node.labels.gunicorn == true]
UpdateConfig:
 Parallelism:   1
 Delay:         30s
 On failure:    continue
 Monitoring Period: 30s
 Max failure ratio: 0
 Update order:      stop-first
RollbackConfig:
 Parallelism:   1
 On failure:    pause
 Monitoring Period: 5s
 Max failure ratio: 0
 Rollback order:    stop-first
ContainerSpec:
 Image:         registry.gitlab.pitchup.com/pitchup/pitchup.com/master:767a4cc0-13456@sha256:1cadc4f4ceaa5a8551636a417bfc0bd76189bdb2afadcd262581c96aeeeb6f64

but on some hosts it doesn't update:

 docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Image}}" | grep gunicorn
pitchup_gunicorn.2.q7fp4u3bkfkz0dr8tpns49vq7                            Up 18 hours (healthy)   registry.gitlab.pitchup.com/pitchup/pitchup.com/master:767a4cc0-13456
pitchup_gunicorn.4.pw9pacn1zhbw0dy29wp9wuygd                            Up 19 hours (healthy)   registry.gitlab.pitchup.com/pitchup/pitchup.com/master:7a464095-13452

Can see it still has the old version.

To fix it I can scale down and scale up and it updates the instances.

The manager:

> docker system info
Containers: 16
 Running: 8
 Paused: 0
 Stopped: 8
Images: 76
Server Version: 18.06.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: n1wd0kbm3e3b38kp2jlnl66fk
 Is Manager: true
 ClusterID: s8l5wsqmc2xxxhja59ycldvw0
 Managers: 3
 Nodes: 6
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 10
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Autolock Managers: false
 Root Rotation In Progress: false
 Node Address: 213.138.116.235
 Manager Addresses:
  213.138.116.235:2377
  213.138.116.236:2377
  213.138.116.237:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d64c661f1d51c48782c9cec8fda7604785f93587
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-128-generic
Operating System: Ubuntu 16.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.757GiB
Name: pitchupbalance.pitchup.com
ID: RP23:QPHI:D4A4:55UO:3XST:GFLN:GCSB:77BN:UE5A:3UPP:HM26:GQEU
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: pitchupmatt
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

on the non manager node that didn't update

> docker system info
Containers: 95
 Running: 21
 Paused: 0
 Stopped: 74
Images: 45
Server Version: 18.06.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: iiihvag5g757ehmcyp9ddkl36
 Is Manager: false
 Node Address: 213.138.116.234
 Manager Addresses:
  213.138.116.235:2377
  213.138.116.236:2377
  213.138.116.237:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d64c661f1d51c48782c9cec8fda7604785f93587
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-128-generic
Operating System: Ubuntu 16.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 88
Total Memory: 377.8GiB
Name: pu-backend1.pitchup.com
ID: 332T:WLCI:NAWD:AYDO:USO3:AC2R:NRUQ:D6IO:Q2QC:DMJR:YCU2:AQTO
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: pitchupmatt
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support