Issue to use global scope when use docker swarm mode and v2plugin

[LipingMao]

Hi Team,

When I trying to use kuryr-libentwork(which is a remote driver) in docker swarm mode, it looks good when I run it with "local" scope in NetworkDriver. But when I run it in "global" capability, it can't works and it show the "swarm" scope. Is it supported run in "global" capability when I use docker swarm mode with v2plugin?

Here is some info:

Docker Version:
# docker version
Client:
 Version:      1.13.1-cs2
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   ad32da7
 Built:        Thu Feb 23 16:40:08 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.13.1-cs2
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   ad32da7
 Built:        Thu Feb 23 16:40:08 2017
 OS/Arch:      linux/amd64
 Experimental: false

Here is the log , it runs normal when it is local scope:

# docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
08da057d698b        bridge              bridge              local
526a0b195163        docker_gwbridge     bridge              local
83f849eb1d60        host                host                local
ouqu3ermkxba        ingress             overlay             swarm
772b4aa8d77b        limao_swarm_kuryr   kuryr:latest        local
1f88364d0ab8        none                null                local

Here is the log, it can't works when I use "global" scope, it shows it is "swarm" scope and can't even show the Driver name(The network name is limao_kuryr1):

# docker network create --driver=kuryr --ipam-driver=kuryr --subnet 10.225.17.0/24  --gateway 10.225.17.1 --ip-range  10.225.17.0/24 -o neutron.net.uuid=55aec833-7838-482f-be9e-b205e5c6f2c4 limao_kuryr1

# docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
3aa3c903c45f        bridge              bridge              local
526a0b195163        docker_gwbridge     bridge              local
83f849eb1d60        host                host                local
ouqu3ermkxba        ingress             overlay             swarm
qazi7mnpl7br        limao_kuryr1                            swarm
1f88364d0ab8        none                null                local

I use global scope, but it shows "swarm" scope here:

# docker network inspect limao_kuryr1
[
    {
        "Name": "limao_kuryr1",
        "Id": "qazi7mnpl7brjxho4kbb62nu7",
        "Created": "0001-01-01T00:00:00Z",
        "Scope": "swarm",
        "Driver": "",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "kuryr:latest",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.225.17.0/24",
                    "IPRange": "10.225.17.0/24",
                    "Gateway": "10.225.17.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Containers": null,
        "Options": null,
        "Labels": null
    }
]

Thanks.

[LipingMao]

Loop @mavenugo @anusha-ragunathan .

I see [1] will be ported in 1.13.1, but looks like it still can't work with scope global plugin v2, I aslo tried to use [2] for test, and the only thing I modify is to change scope from local to global, I get similar problem with that test netplugin.

Can 1.13.1 support v2 network plugin with global scope? Thanks for your help.

[1]https://github.com/docker/docker/pull/30145 [2]https://github.com/mavenugo/test-docker-netplugin

[mavenugo]

@LipingMao in swarm-mode, the global scoped drivers are displayed as swarm scoped to indicate the fact that there is no external kv-store required and swarm-manager handles the creation and management of networks. So, when you see swarm scoped network in swarm-mode, it is a good thing. The driver name should also show up properly. I don't know why it is failing on you.

Can you pls share how you install the plugin and also the output of docker plugin ls ?

[anusha-ragunathan]

@LipingMao : [2] above has not been updated to global/swarm scope.

You can use https://hub.docker.com/r/aragunathan/global-net-plugin as a mock globally scoped network plugin (source at https://github.com/anusha-ragunathan/global-net-plugin)

[LipingMao]

@mavenugo @anusha-ragunathan Thanks for you quick response and support first.

@mavenugo I install the kuryr-libentwork v2plugin by docker plugin install --alias kuryr limao/kuryr-libnetwork-v2plugin . when I configure kuryr-libentwork works in local scope, it works fine, but when I configure it to global scope it can't works. here is the output of docker plugin ls:

# docker plugin ls
ID                  NAME                DESCRIPTION                          ENABLED
f77444226596        kuryr:latest        kuryr-libnetwork plugin for Docker   true

@anusha-ragunathan I tested your repo, it can work in my environment. I compare the code in [1] and [2], In kuryr-libentwork, we has not implyment the following two NetworkDriver API: AllocateNetwork / FreeNetwork

Are these two apis needed when enable globally scoped network plugin in swarm mode? Kuryr-libentwork implyment the apis in [3], but I did not find AllocateNetwork and FreeNetwork in that document. Did I miss anything?

Thanks again for your help @mavenugo and @anusha-ragunathan

[1] https://github.com/mavenugo/test-docker-netplugin [2] https://github.com/anusha-ragunathan/global-net-plugin [3] https://github.com/docker/libnetwork/blob/master/docs/remote.md

[LipingMao]

@mavenugo @anusha-ragunathan

BTW, I can reproduce the same problem by change the scope in [1] from local to global in main.go, and re-build it. Then it will have the same problem with kuryr-libnetwork. If I use that plugin to create network, I can't see DRIVER name.

For example:

# docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
...
qazi7mnpl7br        limao_test                            swarm
...

[1] https://github.com/mavenugo/test-docker-netplugin

[mavenugo]

@LipingMao yes. its a doc issue. PTAL : https://github.com/docker/libnetwork/blob/master/driverapi/driverapi.go and implement the APIs as mentioned in the comments.

Also, PTAL overlay driver as an example of how this is used.

I will fix the docs asap.

[vareti]

+1. Seeing this issue with global scoped plugins in swarm mode.

[anusha-ragunathan]

@LipingMao : Yes, the 2 new APIs need to be implemented for swarm-mode.

@mavenugo : https://github.com/docker/libnetwork/issues/1649 was created a while back :)

[hyunsun]

Hi, I'm getting the following error when I set capability_scope to "global" with the latest Docker-CE (17.03.1-ce). ("local" works fine.) I simply created /etc/kuryr/kuryr.conf file and started a container with kuryr/libnetwork:latest image.

Error response from daemon: rpc error: code = 3 desc = legacy plugin kuryr of type NetworkDriver is not supported in swarm mode

Here's kuryr.conf.

[DEFAULT]
bindir = /usr/libexec/kuryr
capability_scope = global
log_dir = /var/log/kuryr
log_file = kuryr.log

[neutron]
project_name = admin
password = password
username = admin
auth_url = http://10.1.1.161:35357/v2.0
auth_type = password

Any idea?

[anusha-ragunathan]

Legacy network plugins (aka pluginv1) is not supported in swarm mode. This is expected.

[hyunsun]

Do you mean the latest kuryr-libnetwork does not support pluginv2? I also tried https://review.openstack.org/#/c/449038/ this patch to build rootfs and then "docker plugin create" -> "docker plugin enable" but got the same error.

[LipingMao]

@hyunsun from you log message , it looks like you are still using legacy plugin not pluginv2. I'm working on the patch https://review.openstack.org/#/c/449038/ , it works in my local environment, if you have any problem when you try kuryr-libnetwork pluginv2, Can you help to add comments(with reproduce steps) on that patch? or you can ping us in kuryr team irc : openstack-kuryr , my irc: limao , I can debug with you to see what happened.

[hyunsun]

Ah..sorry I'm not familiar with Swarm mode. Do I have to do something to use pluginv2? These are the commands I used. I have the above kuryr.conf under the /etc/kuryr.

$ cd kuryr-libnetwork
$ git review -d 449038
$ contrib/docker/v2plugin/v2plugin_rootfs.sh
$ sudo docker plugin create kuryr/kuryr-libnetwork-v2plugin ./
$ sudo docker plugin ls
ID                  NAME                                     DESCRIPTION                          ENABLED
f57a831f01f1        kuryr/kuryr-libnetwork-v2plugin:latest   kuryr-libnetwork plugin for Docker   false
$ sudo docker plugin enable f57a831f01f1
$ sudo docker plugin ls
ID                  NAME                                     DESCRIPTION                          ENABLED
f57a831f01f1        kuryr/kuryr-libnetwork-v2plugin:latest   kuryr-libnetwork plugin for Docker   true
$ sudo docker network create --driver kuryr test-net
Error response from daemon: rpc error: code = 3 desc = legacy plugin kuryr of type NetworkDriver is not supported in swarm mode

I'll join the irc channel.

[LipingMao]

@hyunsun I think the error here maybe because the driver name should be like this:

# docker network create --driver kuryr/kuryr-libnetwork-v2plugin:latest test-net

I follow your steps with this command can work in my environment.

BTW, the configuration file of kuryr is depend on how you want to deploy kuryr , it can works in vm-nested or direct with neutron lay2 agent, we can discuss this in irc channel if you have problem.

[hyunsun]

@LipingMao It works! Thanks :D

[hyunsun]

Hi @LipingMao, I'm getting another error when I create a service with kuryr driver network.

$ sudo docker network create --driver=kuryr/libnetwork2:latest --ipam-driver=kuryr/libnetwork2:latest --subnet=10.10.10.0/24 --gateway=10.10.10.1 kuryr-net
$ sudo docker service create --name redis --network kuryr-net redis:3.0.6

Error log from the plugin.

2017-07-11 09:21:06.950 12 INFO kuryr_libnetwork.controllers [-] Creating subnetpool with the given pool CIDR
2017-07-11 09:21:20.221 12 ERROR kuryr_libnetwork.utils [-] Unexpected error happened: Subnet with pool f96b5ff6-4065-4eed-9854-3a54122f90c9 does not exist.: KuryrException: Subnet with pool f96b5ff6-4065-4eed-9854-3a54122f90c9 does not exist.
2017-07-11 09:36:28.115 13 INFO kuryr_libnetwork.controllers [-] Subnet already deleted.
2017-07-11 09:36:28.121 13 CRITICAL kuryr [-] Unhandled error: IOError: write error
2017-07-11 09:36:28.121 13 ERROR kuryr IOError: write error
2017-07-11 09:36:28.121 13 ERROR kuryr

Any idea?

BTW, where is the right place for me to post a question?

[LipingMao]

@hyunsun Thanks for you use kuryr-libnetwork, I do not think here is the right place to discuss kuryr-libnetwork related question. kuryr-libnetwork is a openstack sub project, so you can report your problem in https://bugs.launchpad.net/kuryr-libnetwork . And we have irc channel #openstack-kuryr. You can also send mail to openstack mail list : openstack-dev@lists.openstack.org with [kuryr] in your subject. or you can send me mail directly limao@cisco.com if you are not sure how to do. Thanks.

[hyunsun]

@LipingMao thanks for the information.

[rahulbagad]

@mavenugo Is it anyhow possible to ping a container on the overlay from another VM which is participant in swarm?