search

docker-archive / classicswarm

Swarm Classic: a container clustering system. Not to be confused with Docker Swarm which is at https://github.com/docker/swarmkit
Apache License 2.0
5.75k stars 1.08k forks source link

Services stuck in UpdateStatus State of 'updating' for hours #2818

Closed WillNye closed 6 years ago

WillNye commented 6 years ago

Around 1 of 4 releases I encounter an issue where none of the services in our stack will update to the latest images. If I run docker service ls or docker service inspect it shows the correct image but if I run docker ps it will show the old image is being used an that a new container was never spun up. I'm running 3 nodes, each on their own physical host. I drained and pruned each node to make sure that wasn't part of the problem and even manually pulled the images on each node. Here is an example of this.


Service Mode:   Replicated
 Replicas:  3
UpdateStatus:
 State:     updating
 Started:   11 minutes ago
 Message:   update in progress
Placement:
UpdateConfig:
 Parallelism:   2
 Delay:     10s
 On failure:    pause
 Monitoring Period: 5s
 Max failure ratio: 0
 Update order:      stop-first
RollbackConfig:
 Parallelism:   1
 On failure:    pause
 Monitoring Period: 5s
 Max failure ratio: 0
 Rollback order:    stop-first

But running docker ps I can see it didn't update "gunicorn -w 3 -t ..." 39 hours ago Up 39 hours

Version

Client:
 Version:      17.06.0-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:23:31 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.0-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:19:04 2017
 OS/Arch:      linux/amd64
 Experimental: false
nishanttotla commented 6 years ago

@WillNye please open this issue on https://github.com/moby/moby since this is a Swarm mode issue, not related to this repo.