search

docker / for-linux

Docker Engine for Linux
https://docs.docker.com/engine/installation/
751 stars 84 forks source link

docker service update --force SERVICE changes mounts order #893

Open viktor-thell-seal opened 4 years ago

viktor-thell-seal commented 4 years ago

Expected behavior

A forced update of a service should not change the mount order.

Actual behavior

When running a forced update of a service to restart it the order of the mounts change, this causes unexpected service restarts when re-deploying the stack.

Steps to reproduce the behavior

Using this compose-file:

$ cat mount-issue.yml
version: '3.5'

services:
  hello-world:
    image: woahbase/alpine-supervisor
    volumes:
      - type: tmpfs
        target: /tmp
      - /var/log:/test/log
      - /var/log:/test/data/import
      - /var/log:/test/data/blobstore
      - /var/log:/test/data/license
      - /var/log:/test/auth
      - /var/log:/test/release
      - /var/log:/test/data/test-ws/exports
  1. Stack deploy: 
    $ docker stack deploy -c mount-issue.yml mount
  2. Forced service update: 
    $ docker service update --force mount_hello-world
  3. Another stack deploy: 
    $ docker stack deploy -c mount-issue.yml mount

    One can now see that three tasks has been created, although the last deploy should not have resulted in a new task as nothing changed from the first deploy.

    $ docker service ps -q mount_hello-world
4g1tu5w3bndusu4nl4awuyiqe
q9xvpg7paks4fde44o0ejc6iv
752ntfjb3bor9xfua0ozxkete

When comparing the output of docker inspect for the first and the second task one can see that the mounts have changed order:

$ diff docker-inspect-first-task docker-inspect-second-task
3c3
<         "ID": "752ntfjb3bor9xfua0ozxkete",
---
>         "ID": "q9xvpg7paks4fde44o0ejc6iv",
5c5
<             "Index": 392134
---
>             "Index": 392145
7,8c7,8
<         "CreatedAt": "2020-01-09T09:22:39.776079322Z",
<         "UpdatedAt": "2020-01-09T09:25:05.412734209Z",
---
>         "CreatedAt": "2020-01-09T09:25:00.993867387Z",
>         "UpdatedAt": "2020-01-09T09:26:04.369133635Z",
28c28
<                         "Target": "/test/log"
---
>                         "Target": "/test/auth"
33c33
<                         "Target": "/test/data/import"
---
>                         "Target": "/test/data/blobstore"
38c38
<                         "Target": "/test/data/blobstore"
---
>                         "Target": "/test/data/import"
48c48
<                         "Target": "/test/auth"
---
>                         "Target": "/test/data/test-ws/exports"
53c53
<                         "Target": "/test/release"
---
>                         "Target": "/test/log"
58c58
<                         "Target": "/test/data/test-ws/exports"
---
>                         "Target": "/test/release"
80c80
<             "ForceUpdate": 0
---
>             "ForceUpdate": 1
86c86
<             "Timestamp": "2020-01-09T09:25:05.322331467Z",
---
>             "Timestamp": "2020-01-09T09:26:04.344980584Z",
90c90
<                 "ContainerID": "9b2797574a201e962544545e7abf3f617fa023a94832246b653ab921c0279884",
---
>                 "ContainerID": "3e9f2e74a428a058fee8d8593e39cfc9f2f982dd8f13f5e2771e862f40e29813",
135c135
<                     "10.0.19.3/24"
---
>                     "10.0.19.5/24"

However, when comparing the output of docker inspect for the first and the third task one can see that the mounts are back in order:

$ diff docker-inspect-first-task docker-inspect-third-task
3c3
<         "ID": "752ntfjb3bor9xfua0ozxkete",
---
>         "ID": "4g1tu5w3bndusu4nl4awuyiqe",
5c5
<             "Index": 392134
---
>             "Index": 392148
7,8c7,8
<         "CreatedAt": "2020-01-09T09:22:39.776079322Z",
<         "UpdatedAt": "2020-01-09T09:25:05.412734209Z",
---
>         "CreatedAt": "2020-01-09T09:26:00.104479854Z",
>         "UpdatedAt": "2020-01-09T09:26:05.875837575Z",
80c80
<             "ForceUpdate": 0
---
>             "ForceUpdate": 1
86,88c86,88
<             "Timestamp": "2020-01-09T09:25:05.322331467Z",
<             "State": "shutdown",
<             "Message": "shutdown",
---
>             "Timestamp": "2020-01-09T09:26:05.817989009Z",
>             "State": "running",
>             "Message": "started",
90,91c90,91
<                 "ContainerID": "9b2797574a201e962544545e7abf3f617fa023a94832246b653ab921c0279884",
<                 "PID": 0,
---
>                 "ContainerID": "35e681cc36a39c57ab62ded1b33c75021c886ba4eb9dabf4c1d6e995eaf07ff2",
>                 "PID": 21090,
96c96
<         "DesiredState": "shutdown",
---
>         "DesiredState": "running",
135c135
<                     "10.0.19.3/24"
---
>                     "10.0.19.6/24"

Output of docker version:

$ docker version
Client: Docker Engine - Community
 Version:           19.03.3
 API version:       1.40
 Go version:        go1.12.10
 Git commit:        a872fc2f86
 Built:             Tue Oct  8 01:00:44 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.3
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.10
  Git commit:       a872fc2f86
  Built:            Tue Oct  8 00:59:17 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

$ docker info
Client:
 Debug Mode: false

Server:
 Containers: 26
  Running: 24
  Paused: 0
  Stopped: 2
 Images: 343
 Server Version: 19.03.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: 1mq1c0kp9qsfznncxsi5x9adb
  Is Manager: true
  ClusterID: buvaarnh3paslybdpg3qmuavj
  Managers: 1
  Nodes: 1
  Default Address Pool: 10.0.0.0/8  
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 192.168.101.199
  Manager Addresses:
   192.168.101.199:2377
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.0.0-37-generic
 Operating System: Ubuntu 19.04
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 31.08GiB
 Name: viktor-Precision-5530
 ID: 4Y75:LKUS:LXZN:23QS:QIZK:SF5L:ZZK3:2KUC:5EOS:YNJQ:N5OA:YKW3
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  local.seal-software.net:5443
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.) Physical machine

camhorn commented 3 months ago

Observed with non-forced update as well, which seems more problematic.

docker version
Client: Docker Engine - Community
 Version:           26.1.1
 API version:       1.45
 Go version:        go1.21.9
 Git commit:        4cf5afa
 Built:             Tue Apr 30 11:47:53 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.1.1
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.9
  Git commit:       ac2de55
  Built:            Tue Apr 30 11:47:53 2024
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.6.31
  GitCommit:        e377cd56a71523140ca6ae87e30244719194a521
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0