Cannot set multiple gateways

kingsd041 commented 5 years ago

RancherOS Version: (ros os version) 1.5.0-rc1 Where are you running RancherOS? (docker-machine, AWS, GCE, baremetal, etc.) KVM and virtualbox

My rancheros has multiple NICs, I have set up gateways for eth1 and 'eth2'. After the setup is successful, I can't find the default gateway of eth2 on rancheros.

[root@rancher ~]# ros c export
rancher:
  environment:
    EXTRA_CMDLINE: /init
  network:
    interfaces:
      eth0:
        dhcp: true
      eth1:
        address: 10.1.0.41/24
        dhcp: false
        gateway: 10.1.0.1
        mtu: 1500
      eth2:
        address: 10.31.168.85/24
        dhcp: false
        gateway: 10.31.168.1
        mtu: 1500
  state:
    dev: LABEL=RANCHER_STATE
    wait: true
ssh_authorized_keys:
- ssh-rsa ......

[root@rancher ~]# ip route
default via 10.1.0.1 dev eth1
default via 192.168.122.1 dev eth0 src 192.168.122.39 metric 202
10.1.0.0/24 dev eth1 proto kernel scope link src 10.1.0.41
10.31.168.0/24 dev eth2 proto kernel scope link src 10.31.168.85
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.18.0.0/16 dev docker-sys proto kernel scope link src 172.18.42.1 linkdown
192.168.122.0/24 dev eth0 proto kernel scope link src 192.168.122.39 metric 202

Expected results default via 10.31.168.1 dev eth2

niusmallnan commented 5 years ago

Setting up multiple default gateways does seem to make no sense.

Jason-ZW commented 5 years ago

Analysis

Judging from the code intent, we want to add multiple default gateways when configuring different routes with multiple network cards. But we have the following error when adding the second default gateway, this error is the cause of the failed addition:

file exists

This error occurs because a default route gateway without metric property already exists in the routing table, others route gateway can not be added without metric property.

default via 192.168.1.1 dev eth1  # The root cause
default via 10.0.2.2 dev eth0 src 10.0.2.15 metric 203
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15 metric 203
172.30.1.0/24 dev eth2 proto kernel scope link src 172.30.1.100
192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.100

Test

Prepare: Add a route so that `172.30.1.0/24` network can reachable via `eth2`.

default via 192.168.1.1 dev eth1  # The root cause
default via 10.0.2.2 dev eth0 src 10.0.2.15 metric 203
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15 metric 203
172.30.1.0/24 dev eth2 proto kernel scope link src 172.30.1.100
192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.100

$ ip route add 172.30.1.0/24 dev eth2

Step 1. Execute the test code add another default route gateway, same error:

Error occurs: file exists

Step2. Remove the default gateway `192.168.1.1` which without `metric` property and re-add the default gateway with `metric` property:

$ route del default gw 192.168.1.1
$ ip route add default via 192.168.1.1 src 192.168.1.100 metric 204

default via 10.0.2.2 dev eth0 src 10.0.2.15 metric 203
default via 192.168.1.1 dev eth1 src 192.168.1.100 metric 204
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15 metric 203
172.30.1.0/24 dev eth2 proto kernel scope link src 172.30.1.100
192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.100

Step3. Execute the test code add another default route gateway, success:

default via 172.30.1.1 dev eth2
default via 10.0.2.2 dev eth0 src 10.0.2.15 metric 203
default via 192.168.1.1 dev eth1 src 192.168.1.100 metric 204
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15 metric 203
172.30.1.0/24 dev eth2 proto kernel scope link src 172.30.1.100
192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.100

rancher / os