dockerng port_binding to udp port stopped working after upgrade to 2015.8.3

[provonet]

We are using the dockerng module to deploy several docker containers including a container which contains a consul instance which we use for name resolution. With version 2015.8.1 we could forward dns traffic from the host (dnsmasq) to this container. But with the same salt state this stopped working with version 2015.8.3.

Please see below the port_binding config we use:

    - port_bindings:
        - 8300:8300
        - 8301:8301
        - 8301:8301/udp
        - 8302:8302
        - 8302:8302/udp
        - 8400:8400
        - 8500:8500
        - 8600:8600
        - 8600:8600/udp

with version 2015.8.1 this results in this container config:

7779d0fe4ff5        deltaforce.io/consul_server:latest   "/usr/sbin/runit.sh"   31 seconds ago      Up 29 seconds       0.0.0.0:8300-8302->8300-8302/tcp, 0.0.0.0:8400->8400/tcp, 0.0.0.0:8500->8500/tcp, 0.0.0.0:8301-8302->8301-8302/udp, 0.0.0.0:8600->8600/tcp, 0.0.0.0:8600->8600/udp   consul_server

but with version 2015.8.3:

4c54d32608ef        deltaforce.io/consul_server:0.201512.68   "/usr/sbin/runit.sh"   36 seconds ago      Up 35 seconds       0.0.0.0:8300-8302->8300-8302/tcp, 0.0.0.0:8400->8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp, 0.0.0.0:8600->8600/tcp, 8600/udp   consul_server

[jfindlay]

@provonet, thanks for the report. @ticosax, do you have any ideas on this one?

[ticosax]

@provonet please provide details about your docker-py version

My first guess is the switch to host_config support. Where arguments are passed on creation time rather than starting time of the container.

https://docker-py.readthedocs.org/en/latest/hostconfig/

We have a unit test suite, I'd rely on it to reproduce your use case within a test case. Then the fix will be easier to spot.

https://github.com/saltstack/salt/blob/develop/tests/unit/modules/dockerng_test.py

[provonet]

pip freeze | grep docker
docker-py==1.4.0

I tried 1.5.0 as well but that gives me the same results.

please find below the output of salt-call --versions:

salt-call --versions
Salt Version:
           Salt: 2015.8.3

Dependency Versions:
         Jinja2: 2.7.2
       M2Crypto: Not Installed
           Mako: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.7.0
         Python: 2.7.5 (default, Nov 20 2015, 02:00:19)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: Not Installed

System Versions:
           dist: centos 7.0.1406 Core
        machine: x86_64
        release: 3.10.0-123.8.1.el7.x86_64
         system: CentOS Linux 7.0.1406 Core

[jgmchan]

I'm using the same versions (Salt: v2015.8.3 and docker-py: 1.5.0) and am experiencing the same issue after an upgrade from v2015.8.1. This is what I see in the exposed ports of the docker container:

vagrant@minion:~$ sudo docker inspect --format '{{ .Config.ExposedPorts | json}}' logstash | jq .
{
  "5514/udp/tcp": {},
  "5514/tcp": {},
  "5000/udp/tcp": {},
  "5000/tcp": {}
}

The state looks like this:

logstash-install:
  dockerng.running:
    - name: "logstash"
    - image: "logstash:2"
    - port_bindings: 
      - '514:5514/tcp'
      - '514:5514/udp'
      - '5000:5000/tcp'
      - '5000:5000/udp'

Looks like the protocol is not being passed from the port_bindings.

[jgmchan]

You can work around this by specifying the ports explicitly:

logstash-install:
  dockerng.running:
    - name: "logstash"
    - image: "logstash:2"
    - port_bindings: 
      - '514:5514/tcp'
      - '514:5514/udp'
      - '5000:5000/tcp'
      - '5000:5000/udp'
    - ports:
      - '5514/tcp'
      - '5514/udp'
      - '5000/tcp'
      - '5000/udp'

Some jinja magic like {{ port.split(':')[1] }} makes it a little cleaner but it would be good to not need to do this.

[provonet]

with above mentioned workarround we where able to upgrade to 2015.8.4

[ticosax]

@provonet please can you paste a command that I can reproduce the issue and emphasis what is wrong ?

[provonet]

See my original post from 14/12 or the comment from @jgmchan We use the workarround suggested by @jgmchan which works fine... but is a bit bloated. so for forwarding tcp ports is is enough to only configure them as port_bindings, but for udp ports it is required to set them in ports: as well. With 2015.8.1 that was not needed.

[dverbeek84]

Workaround is not working any more in 2015.8.8! So it is even not possible to open a UDP port with dockerng. I have tried all options, but nothing is working...

docker-py: 1.7.2
docker: 1.10.2

Salt Version:
           Salt: 2015.8.8

Dependency Versions:
         Jinja2: 2.7.2
       M2Crypto: 0.21.1
           Mako: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.7.0
         Python: 2.7.5 (default, Nov 20 2015, 02:00:19)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libgit2: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.7
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: Not Installed

System Versions:
           dist: centos 7.2.1511 Core
        machine: x86_64
        release: 3.10.0-229.14.1.el7.x86_64
         system: CentOS Linux 7.2.1511 Core

If i look in the debug i see something that may cause the issue:

[DEBUG   ] dockerng.running: Analysis of container 'pritunl' after creation/replacement reveals the following changes still need to be made: 
    {'ports': 
      {
        'new': ['5000/tcp', '5000/udp', '5000/udp', '5514/tcp', '5514/udp', '5514/udp'], 
        'old': [u'5000/tcp', u'5000/udp', u'5000/udp/tcp', u'5514/tcp', u'5514/udp', u'5514/udp/tcp']
      }
    }

This seems wrong 5000/udp/tcp

[ticosax]

I'm willing to help if someone can help me in return. I'm just asking for a reproducible use case. BTW I believe this has been fixed by #32015 @dverbeek84 Please, can you give it a try ?

[dverbeek84]

@ticosax i have tested branch 2015.8 and i can confirm that #32015 fixed this issue.

[ticosax]

@dverbeek84 thanks

[jfindlay]

@ticosax, @dverbeek84, thanks for fixing and confirming. If anyone else encounters this problem, we can reopen the issue.

[zmes50416]

I encouter this problem on 2017.07 salt version

statsd:
  docker_container.running:
    - image: atlassianlabs/gostatsd
    - port_bindings:
      - '8125:8125/udp'

I can fix it by adding the workaround mention above - ports: 8125/udp

python  
docker (2.4.2)

Salt Version:
           Salt: 2017.7.0

Dependency Versions:
           cffi: 1.10.0
       cherrypy: unknown
       dateutil: Not Installed
      docker-py: 2.4.2
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: 2.17
       pycrypto: 2.6.1
   pycryptodome: 3.4.3
         pygit2: Not Installed
         Python: 2.7.5 (default, Nov  6 2016, 00:28:07)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.3.1611 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-514.16.1.el7.x86_64
         system: Linux
        version: CentOS Linux 7.3.1611 Core

[MorphBonehunter]

Problem still exists in:

Salt Version:
           Salt: 2017.7.2

Dependency Versions:
           cffi: 1.11.2
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: 2.7.0
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.24.0
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.0
   mysql-python: Not Installed
      pycparser: 2.18
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.14 (default, Sep 20 2017, 01:25:59)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.3
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.2
            ZMQ: 4.2.2

System Versions:
           dist:   
         locale: UTF-8
        machine: x86_64
        release: 4.9.75-1-lts
         system: Linux
        version: Not Installed

Testcase:

docker_container_image_test:
    docker_container.running:
        - name: testcase
        - image: busybox:latest
        - port_bindings: "1234:1234/udp"
        - command: "sleep 300"

creates following container:

docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
019743b15cf7        busybox:latest      "sleep 300"         59 seconds ago      Up 57 seconds       1234/udp            testcase

nothing is listening on the host:

netstat -plun | grep 1234 || echo $?
1

docker inspect:

<snip>
            "PortBindings": {
                "1234/udp": [
                    {
                        "HostIp": "",
                        "HostPort": "1234"
                    }
                ]
            },
<snip>
            "ExposedPorts": {
                "1234/udp/tcp": {}
            },
<snip>
            "Ports": {
                "1234/udp": null
            },
<snip>

so change the testcase:

docker_container_image_test:
    docker_container.running:
        - name: testcase
        - image: busybox:latest
        - port_bindings: "1234:1234/udp"
        - ports: "1234/udp"
        - command: "sleep 300"

salt generates following changes:

     Changes:   
              ----------
              container:
                  ----------
                  Config:
                      ----------
                      ExposedPorts:
                          ----------
                          new:
                              ----------
                              1234/udp:
                                  ----------
                          old:
                              ----------
                              1234/udp/tcp:
                                  ----------

after that following container is running:

CONTAINER ID        IMAGE               COMMAND             CREATED              STATUS              PORTS                    NAMES
d34f7cb5c8c2        busybox:latest      "sleep 300"         About a minute ago   Up About a minute   0.0.0.0:1234->1234/udp   testcase

listen on the host:

netstat -plun | grep 1234 || echo $?
udp6       0      0 :::1234                 :::*                                31891/docker-proxy

docker inspect:

<snip>
            "PortBindings": {
                "1234/udp": [
                    {
                        "HostIp": "",
                        "HostPort": "1234"
                    }
                ]
            },
<snip>
            "ExposedPorts": {
                "1234/udp": {}
            },
<snip>
            "Ports": {
                "1234/udp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "1234"
                    }
                ]
            },
<snip>

@jfindlay or @ticosax would you mind to reopen the case?