failed to create endpoint on network tes23 because network id not found

[wangweihong]

I use two nodes to build overlay network. One node can inspect/connect/disconnect overlay well, but the other one can only inspect network's info , When I try to connect an container to this network: It fails for :

kiongf@youruncloud:~/k8s/project$ sudo docker run -d --net=tes23 busybox sleep 1000 ba0a0064f1de046f8b25dfe67c6bad2b22f3f972cfa49a49bcbf0b0cfd006c7d docker: Error response from daemon: failed to create endpoint backstabbing_archimedes on network tes23: network id "50a3fcedfe64b4834564743ddb43e03ee12409c002ce3aa6444dacbfc4bfa300" not found.

I can inspect this network on this node :

kiongf@youruncloud:~/k8s/project$ docker network inspect tes23 WARNING: Error loading config file:/home/kiongf/.docker/config.json - stat /home/kiongf/.docker/config.json: permission denied [ { "Name": "tes23", "Id": "50a3fcedfe64b4834564743ddb43e03ee12409c002ce3aa6444dacbfc4bfa300", "Scope": "global", "Driver": "overlay", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": null, "Config": [ { "Subnet": "10.12.0.0/16" } ] },

This is my docker info :

kiongf@youruncloud:~/k8s/project$ docker info WARNING: Error loading config file:/home/kiongf/.docker/config.json - stat /home/kiongf/.docker/config.json: permission denied Containers: 11 Running: 0 Paused: 0 Stopped: 11 Images: 91 Server Version: 1.11.2 Storage Driver: aufs Root Dir: /var/lib/docker/aufs Backing Filesystem: extfs Dirs: 162 Dirperm1 Supported: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: overlay bridge null host Kernel Version: 4.4.0-34-generic Operating System: Ubuntu 16.04 LTS OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 7.756 GiB Name: youruncloud ID: JX2L:QY7S:7GTK:QOGL:Y6PM:JIXV:4OI6:CYGK:JYBV:SO5P:CDQR:YIMS Docker Root Dir: /var/lib/docker Debug mode (client): false Debug mode (server): true File Descriptors: 17 Goroutines: 57 System Time: 2016-08-18T09:33:24.213378763+08:00 EventsListeners: 0 Registry: https://index.docker.io/v1/ WARNING: No swap limit support Cluster store: etcd://192.168.4.11:2379 Cluster advertise: 192.168.4.11:2376

[BSWANG]

I meet this issue too , probably you use mix version of docker engine in one cluster , docker 1.12.x overlay network driver not compatible to before .

[srhb]

I've just experienced this as well (mixing a 1.11.something daemon with 17.03 ce.)

Prompted by the comment from @BSWANG a useful workaround (with mostly unknown consequences) was to remove all containers on the network, destroy the network and recreate it from one of the old daemons.

[akamac]

I am experiencing a similar issue while running the same version of docker across nodes.

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 18.09.3
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: journald
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
runc version: 09c8266bf2fcf9519a651b04ae54c967b9ab86ec
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-39-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 961.5MiB
Name: docker-vx-2
ID: BDHW:7P75:QN64:FYG3:WVEQ:VQ4P:D5MU:2KUP:DXRG:V4B6:3DMG:IJZB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Cluster Store: consul://localhost:8500
Cluster Advertise: 10.254.192.12:2376
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: true
Product License: Community Engine

WARNING: No swap limit support

I have configured docker to use external kv store (consul) and created ipvlan network with --scope global:

On each node: docker network create --config-only --opt macvlan_mode=bridge --opt ipvlan_mode=l2 --opt parent=eth1.1605 --subnet 10.254.196.0/24 --gateway 10.254.196.1 --ip-range=10.254.196.224/27 ipvlan-conf

On one node: docker network create --driver macvlan --scope global --config-from ipvlan-conf ipvlan-net

I can run containers on the node where ipvlan-net network was created, but fail on others with docker: Error response from daemon: failed to create endpoint frosty_chebyshev on network ipvlan-net: network id "27654ac5c33819eb64c6e329def9bb7ebf2b5b0e221032160761b6f1596f7893" not found., while docker network ls/inspect sees ipvlan-net network, as does containers attached to it:

NETWORK ID          NAME                DRIVER              SCOPE
a2abc68e44a6        bridge              bridge              local
34c1b7a4e51c        docker_gwbridge     bridge              local
0b89e9dfaaeb        host                host                local
8394ed5d1bf9        ipvlan-conf         null                local
27654ac5c338        ipvlan-net          macvlan             global
f7b6c7174fe1        none                null                local

[
    {
        "Name": "ipvlan-net",
        "Id": "27654ac5c33819eb64c6e329def9bb7ebf2b5b0e221032160761b6f1596f7893",
        "Created": "2019-03-07T07:15:09.070697339-08:00",
        "Scope": "global",
        "Driver": "macvlan",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "10.254.196.0/24",
                    "IPRange": "10.254.196.224/27",
                    "Gateway": "10.254.196.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": "ipvlan-conf"
        },
        "ConfigOnly": false,
        "Containers": {
            "ep-d715401e7936392e346c2df7b687bc0d4978e61e2d225b067702cecdce95f6c2": {
                "Name": "vigorous_curran",
                "EndpointID": "d715401e7936392e346c2df7b687bc0d4978e61e2d225b067702cecdce95f6c2",
                "MacAddress": "02:42:0a:fe:c4:e0",
                "IPv4Address": "10.254.196.224/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "ipvlan_mode": "l2",
            "macvlan_mode": "bridge",
            "parent": "eth1.1605"
        },
        "Labels": {}
    }
]

@mavenugo Any ideas? Debug logging does not help to find a root cause.

The goal of this setup is to get IPAM working across the nodes for ipvlan network without swarm (will use Nomad).