failed to add interface

ghost commented 4 years ago

[x] This is a bug report
[ ] This is a feature request
[x] I searched existing issues before opening this one

Expected behavior

docker swarm will update the service I will get text in docker service ps SERVICE in column CURRENT STATE "Running 2 minutes ago"

Actual behavior

Sometimes i get an error: "Failed 22 minutes ago "starting container failed: OCI runtime create failed: container_linux.go:346: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 0 caused \\"error running hook: exit status 1, stdout: , stderr: time=\\\\"2019-11-12T19:09:09+07:00\\\\" level=fatal msg=\\\\"failed to add interface veth44d4290 to sandbox: failed to set link up: device or resource busy\\\\"\\n\\"\"": unknown"" Sometimes everything goes fine

Steps to reproduce the behavior

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.4
 API version:       1.40
 Go version:        go1.12.10
 Git commit:        9013bf583a
 Built:             Fri Oct 18 15:52:34 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.4
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.10
  Git commit:       9013bf583a
  Built:            Fri Oct 18 15:51:05 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 9
  Running: 8
  Paused: 0
  Stopped: 1
 Images: 30
 Server Version: 19.03.4
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: xxgm910o15wodmpkbdaj56lov
  Is Manager: true
  ClusterID: e2kr0peov0mor0s22gtclvnai
  Managers: 3
  Nodes: 3
  Default Address Pool: 10.0.0.0/8
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: REDACTED
  Manager Addresses:
   REDACTED:2377
   REDACTED:2377
   REDACTED:2377
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.9.0-8-amd64
 Operating System: Debian GNU/Linux 9 (stretch)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.863GiB
 Name: nodehostname01
 ID: RZU4:YORE:LILU:ZVAU:RNC6:76ZO:GZOK:UXYG:HBGA:PP3N:DNPN:ZQZG
 Docker Root Dir: /var/storage/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  REDACTED
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.) Selfhosted VM on KVM Debian 9 Networking driver - macvlan

andrewhsu commented 4 years ago

@xupyp1 can you provide the commands you ran to get the error?

ghost commented 4 years ago

We usually use the command docker service update --detach --force --update-failure-action=rollback --with-registry-auth --image=$IMAGE --config-rm OLD --config-add NEW $SERVICENAME

Last command was: docker service update --detach --force --update-failure-action=rollback --with-registry-auth --image=REDACTED --config-rm REDACTED.00000002 --config-add source=REDACTED.properties.00000002,target=REDACTED --config-rm REDACTED.00000007 --config-add source=REDACTED.00000007,target=REDACTED --config-rm REDACTED.00000026 --config-add source=REDACTED.00000026,target=REDACTED --update-parallelism=1 --update-delay=60s REDACTED

docker / for-linux