Docker Compose + Container network + internal bridge network cannot express an open port on the container network

[corrieb]

User Statement:

As a container developer, network isolation and firewalls are important to me. I want to create a simple classic app in which containers talk on an internal bridge network and one of the containers exposes a port on an external container network. I want to use Compose to achieve this. I do not want to use port mapping for reasons of performance and SPOF.

Details:

This is not a complex scenario. However, it is a more realistic real-world scenario than the ones we've been testing and documenting. One of our core value propositions is the ability to use container networks and we've recently introduced a change where ports on a container network need to be explicitly opened using -p. The problem is, Docker Compose doesn't seem to be able to express that. Compose seems to assume you always want port mapping and regardless of the ordering in the Compose file, it will not pass -p to the external network.

This is partly a limitation in the way that the multiple network support in compose was designed - there is no obvious way to assign port mappings to a particular network.

I've spent hours trying to figure out a workaround here. I've raised a issue with the compose team here: https://github.com/docker/compose/issues/5042 to at least see if there's a way to influence the ordering.

Bottom line is that as it stands, docker-compose does not work for a very important scenario. See example compose file here:

version: '2'

services:
   db:
     image: mysql:5.7
     command: --datadir=/var/lib/mysql/data
     volumes:
       - db-data:/var/lib/mysql
     networks:
       - db-net
     environment:
       MYSQL_ROOT_PASSWORD: somewordpress
       MYSQL_DATABASE: wordpress
       MYSQL_USER: wordpress
       MYSQL_PASSWORD: wordpress

   wordpress:
     depends_on:
       - db
     image: wordpress:latest
     ports:
       - "80"
     volumes:
       - html-data:/var/www/html
     networks:
       - db-net
       - outside
     environment:
       WORDPRESS_DB_HOST: db:3306
       WORDPRESS_DB_USER: wordpress
       WORDPRESS_DB_PASSWORD: wordpress

volumes:
    db-data:
       driver: "vsphere"
       driver_opts:
          Capacity: "4G"
          VolumeStore: "default"
    html-data:
       driver: "vsphere"
       driver_opts:
          Capacity: "2G"
          VolumeStore: "default"

networks:
    outside:
       external:
          name: ExternalNetwork
    db-net:
       internal: true

Acceptance Criteria:

There's nothing obvious we can fix on our side at present. The best workaround I can suggest is that we use the opaque "driver_opts" https://docs.docker.com/compose/compose-file/compose-file-v2/#driver_opts-1 value on the network definition to specify what ports should be opened, but that will require us to change our API. Let's wait and see what response we get from Docker.

Frankly that's a less ugly option that just saying "compose doesn't work with container networks"

[corrieb]

I've marked this as an investigation for now, but this will get moved to being a bug if we get no workaround

[hickeng]

@corrieb What do you think about using specification of 0.0.0.0:port to mean expose via container-network in addition to NAT if container-network trust level is published?

0.0.0.0 is regular docker default for bind to everything, however VCH is inherently opinionated about where ports are exposed - it's via the public network role if NATing, and via the specified container-network otherwise. As such I've stopped using 0.0.0.0 in returned data (see #6641) so that we can return useful addresses which means it could be leveraged to mean bind to everything for VCH as well. I've not through through potential UX confusion, just soliciting input.

[ghost]

For anyone looking for temporary workaround: sharing unix socket between containers and mapping to port with socat worked perfectly for me.

[hickeng]

Addressing the ordering issues for port forwarding via container-networks should also be addressed when working on this issue. See the following from #6557:

Additional work for the port forwarding: associate forwarded ports with an external scope network irrespective of network connect ordering (see #5776) so long as network is configured with Published or Open trust levels. Not going to be in this PR.