[BUG] eth0 default route priority define outbound connection

mrpops2ko commented 1 week ago

Description

hi i've been reading through all the documentation and i'm struggling on how to define which networks in docker compose become the default gateway

from googling i can find various resources which say that its;

in the order in which it is defined in the compose file (but this seems not to be true)
that it is based upon the alphabetic naming of the networks, so A will begin before B etc (this seems to be true for customer docker networks, but it doesn't work similarly when combined with ipvlan networks)

I stumbled across a few different issues here which talked about priority and i thought that i could use that to define which network becomes the default outbound gateway but that doesn't appear to be the case either, i'm not sure what priority does - from reading the bug thread it mentioned it was something in relation to mac addresses.

Steps To Reproduce

    networks:
      intraNW:
        priority: 10
      br0:
        priority: 30
        ipv4_address: 192.168.1.145
      a-traefik-public:
        priority: 20

take for example this, br0 is an ipvlan network and the other 2 are custom docker networks

my use case scenario is that i want to have as eth0 the br0 interface. the reason for this is because i wish to do split tunnelling on the router and in order to do that i need ipvlan.

it works fine if i remove the other custom networks, but i can't for the life of me seem to be able to get the ipvlan br0 network to become the default eth0 network whilst using docker custom networks alongside it

one possible solution to this seems to be creating a container init script and running that on load, which will change the default route to the desired one but i think this is not a very clean solution compared to a native implementation which docker compose

Compose Version

Docker Compose version v2.29.2

Docker Environment

Client:
 Version:    24.0.9
 Context:    default
 Debug Mode: false
 Plugins:
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.2
    Path:     /usr/local/lib/docker/cli-plugins/docker-compose

Server:
 Containers: 79
  Running: 50
  Paused: 0
  Stopped: 29
 Images: 209
 Server Version: 24.0.9
 Storage Driver: btrfs
  Btrfs:
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7c3aca7a610df76212171d200ca3811ff6096eb8
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.99-Unraid
 Operating System: Slackware 15.0 x86_64 (post 15.0 -current)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 78.56GiB
 Name: alexandria
 ID: xxx
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: hvrpride
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: No swap limit support

Anything else?

No response

mrpops2ko commented 1 week ago

these are some of the other threads on this

https://github.com/docker/compose/issues/4645 https://github.com/moby/moby/pull/43518 https://github.com/docker/compose/issues/11229 https://github.com/docker/compose/issues/8561

this seems like a longstanding issue

ndeloof commented 6 days ago

priority indeed defines which network is connected first when creating container with moby engine, but AFAIK the actual engine implementation does not use it to define default gateway, but does some alphanumeric sorting @akerouanton might better know the status of this in latest engine codebase

akerouanton commented 4 days ago

@mrpops2ko Could you share you network definitions please?

mrpops2ko commented 4 days ago

@mrpops2ko Could you share you network definitions please?

root@alexandria:~# docker network ls
NETWORK ID     NAME                       DRIVER    SCOPE
e05c170d5559   a-traefik-public           bridge    local
32974c0c0378   br0                        ipvlan    local
f16259b5cce2   bridge                     bridge    local
e09d59b70229   host                       host      local
e75d14eecf63   none                       null      local

root@alexandria:~# docker network inspect e05c170d5559
[
    {
        "Name": "a-traefik-public",
        "Id": "e05c170d55594e375f582aba5d35c06b34d4d86774cd86cf750a39de269faca1",
        "Created": "2024-05-15T00:39:20.032951453+01:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "0350ae8ef3d8a2b700ccec71d028725fe3e2684a784dbbd3163bc099e3d879ac": {
xxx
            }
        },
        "Options": {},
        "Labels": {}
    }
]

root@alexandria:~# docker network inspect 32974c0c0378
[
    {
        "Name": "br0",
        "Id": "32974c0c0378747271ab726429616985d953d9a2bac744eb94cf231a50ec8199",
        "Created": "2024-09-14T13:42:25.513278678+01:00",
        "Scope": "local",
        "Driver": "ipvlan",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "192.168.0.0/20",
                    "Gateway": "192.168.1.1",
                    "AuxiliaryAddresses": {
                        "server": "192.168.1.3"
                    }
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "096fe31bbbf0d98864d108c1f6adab9cd9800451ef0ff8b76af75df77bed345e": {
xxx
            }
        },
        "Options": {
            "parent": "br0"
        },
        "Labels": {}
    }
]

the other networks are all created upon docker compose startup

docker / compose