Octavia: Connect provider network and lb-mgmt-net

oomichi commented 5 years ago

外向けのネットワーク provider と Octavia 管理用ネットワーク lb-mgmt-net を疎通させる。

https://github.com/oomichi/try-kubernetes/issues/68#issuecomment-528601993

oomichi commented 5 years ago

環境のクリーンアップ

Routerの削除 → 失敗

$ openstack router delete router01
Failed to delete router with name or ID 'router01': Unable to delete Router for openstack.network.v2.router.Router(status=ACTIVE, external_gateway_info={u'network_id': u'bfd9fd43-c9b4-43ad-bb67-930c674f2605', u'enable_snat': True, u'external_fixed_ips': [{u'subnet_id': u'43ed897b-3c10-4d5c-8f6d-263edcd817c7', u'ip_address': u'192.168.1.115'}]}, availability_zone_hints=[], availability_zones=[], description=, tags=[], tenant_id=682e74f275fe427abd9eb6759f3b68c5, created_at=2019-05-07T02:22:48Z, admin_state_up=True, distributed=False, updated_at=2019-05-07T02:25:43Z, flavor_id=None, routes=[], ha=False, revision=4, id=8f079f83-1585-4783-b188-2f62141c1b5c, name=router01)
1 of 1 routers failed to delete.

Router から subnet を削除 → Router削除が成功

$ openstack router remove subnet router01 lb-mgmt-subnet
$ openstack router remove subnet router01 provider
$ openstack router delete router01

Octavia管理用サブネットを削除

$ openstack subnet delete lb-mgmt-subnet

oomichi commented 5 years ago

Octavia 公式（予定）の手順で構築する

https://review.opendev.org/#/c/672842/ の手順

      $ OCTAVIA_MGMT_SUBNET=172.16.0.0/12
      $ OCTAVIA_MGMT_SUBNET_START=172.16.0.100
      $ OCTAVIA_MGMT_SUBNET_END=172.16.31.254
      $ OCTAVIA_MGMT_PORT_IP=172.16.0.2

      $ openstack network create lb-mgmt-net
      $ openstack subnet create --subnet-range $OCTAVIA_MGMT_SUBNET \
        --allocation-pool start=$OCTAVIA_MGMT_SUBNET_START,\
        end=$OCTAVIA_MGMT_SUBNET_END --network lb-mgmt-net lb-mgmt-subnet

      $ SUBNET_ID=$(openstack subnet show lb-mgmt-subnet -f value -c id)
      $ PORT_FIXED_IP="--fixed-ip subnet=$SUBNET_ID,ip-address=$OCTAVIA_MGMT_PORT_IP"

      $ MGMT_PORT_ID=$(openstack port create --security-group \
        lb-health-mgr-sec-grp --device-owner Octavia:health-mgr \
        --host=$(hostname) -c id -f value --network lb-mgmt-net \
        $PORT_FIXED_IP octavia-health-manager-listen-port)

      $ MGMT_PORT_MAC=$(openstack port show -c mac_address -f value \
        $MGMT_PORT_ID)
      $ MGMT_PORT_IP=$(openstack port show -f value -c fixed_ips \
        $MGMT_PORT_ID | awk '{FS=",| "; gsub(",",""); gsub("'\''",""); \
        for(i = 1; i <= NF; ++i) {if ($i ~ /^ip_address/) {n=index($i, "="); \
        if (substr($i, n+1) ~ "\\.") print substr($i, n+1)}}}')

      $ sudo ip link add o-hm0 type veth peer name o-bhm0
      $ NETID=$(openstack network show lb-mgmt-net -c id -f value)
      $ BRNAME=brq$(echo $NETID|cut -c 1-11)
      $ sudo brctl addif $BRNAME o-bhm0
      $ sudo ip link set o-bhm0 up

      $ sudo ip link set dev o-hm0 address $MGMT_PORT_MAC
      $ sudo iptables -I INPUT -i o-hm0 -p udp --dport 5555 -j ACCEPT
      $ sudo dhclient -v o-hm0

oomichi commented 5 years ago

最後の sudo dhclient -v o-hm0 を実行したタイミングでネットワークが不通になる。 devstack で指定している下記の内容の設定ファイルを指定しても同様。

 request subnet-mask,broadcast-address,interface-mtu;
 do-forward-updates false;

oomichi commented 5 years ago

下記のように api ノードではない場合に実行することになっている

500 function octavia_start {
501
502     if  ! ps aux | grep -q [o]-hm0 && [ $OCTAVIA_NODE != 'api' ] ; then
503         sudo dhclient -v o-hm0 -cf $OCTAVIA_DHCLIENT_CONF
504     fi

oomichi commented 5 years ago

docker0 のアドレスが 172.17.0.1 となっている。このネットワークアドレスは docker のデフォルト。

docker0   Link encap:Ethernet  HWaddr 02:42:66:9c:c4:5e
          inet addr:172.17.0.1  Bcast:172.17.255.255  Mask:255.255.0.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

これが 172.16.0.0/12 の lb-mgmt-subnet のアドレスと重複している。重複しないように変更したほうが良さそう。 devstack での設定値を調べてみる。 octavia/devstack/settings

OCTAVIA_MGMT_SUBNET=${OCTAVIA_MGMT_SUBNET:-"192.168.0.0/24"}
OCTAVIA_MGMT_SUBNET_START=${OCTAVIA_MGMT_SUBNET_START:-"192.168.0.2"}
OCTAVIA_MGMT_SUBNET_END=${OCTAVIA_MGMT_SUBNET_END:-"192.168.0.200"}

192.168.1.0/24は使っているが、192.168.0.0/24 は使っていないのでこれを利用する。

      $ OCTAVIA_MGMT_SUBNET=192.168.0.0/24
      $ OCTAVIA_MGMT_SUBNET_START=192.168.0.100
      $ OCTAVIA_MGMT_SUBNET_END=192.168.0.200
      $ OCTAVIA_MGMT_PORT_IP=192.168.0.2

      $ openstack network create lb-mgmt-net
      $ openstack subnet create --subnet-range $OCTAVIA_MGMT_SUBNET \
        --allocation-pool start=$OCTAVIA_MGMT_SUBNET_START,end=$OCTAVIA_MGMT_SUBNET_END \
        --network lb-mgmt-net lb-mgmt-subnet

      $ SUBNET_ID=$(openstack subnet show lb-mgmt-subnet -f value -c id)
      $ PORT_FIXED_IP="--fixed-ip subnet=$SUBNET_ID,ip-address=$OCTAVIA_MGMT_PORT_IP"

      $ MGMT_PORT_ID=$(openstack port create --security-group \
        lb-health-mgr-sec-grp --device-owner Octavia:health-mgr \
        --host=$(hostname) -c id -f value --network lb-mgmt-net \
        $PORT_FIXED_IP octavia-health-manager-listen-port)

      $ MGMT_PORT_MAC=$(openstack port show -c mac_address -f value  $MGMT_PORT_ID)
      $ MGMT_PORT_IP=$(openstack port show -f value -c fixed_ips \
        $MGMT_PORT_ID | awk '{FS=",| "; gsub(",",""); gsub("'\''",""); \
        for(i = 1; i <= NF; ++i) {if ($i ~ /^ip_address/) {n=index($i, "="); \
        if (substr($i, n+1) ~ "\\.") print substr($i, n+1)}}}')

      $ sudo ip link add o-hm0 type veth peer name o-bhm0
      $ NETID=$(openstack network show lb-mgmt-net -c id -f value)
      $ BRNAME=brq$(echo $NETID|cut -c 1-11)
      $ sudo brctl addif $BRNAME o-bhm0
      $ sudo ip link set o-bhm0 up

      $ sudo ip link set dev o-hm0 address $MGMT_PORT_MAC
      $ sudo iptables -I INPUT -i o-hm0 -p udp --dport 5555 -j ACCEPT
      $ sudo dhclient -v o-hm0

oomichi commented 5 years ago

sudo ip link set o-bhm0 up 実行直前の状態

13: o-bhm0@o-hm0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop master brqa1859074-54 state DOWN mode DEFAULT group default qlen 1000
    link/ether 46:50:af:d9:5f:75 brd ff:ff:ff:ff:ff:ff
14: o-hm0@o-bhm0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether c2:c4:57:48:86:c0 brd ff:ff:ff:ff:ff:ff

sudo ip link set o-bhm0 up 実行後の状態

13: o-bhm0@o-hm0: <NO-CARRIER,BROADCAST,MULTICAST,UP,M-DOWN> mtu 1500 qdisc noqueue master brqa1859074-54 state LOWERLAYERDOWN mode DEFAULT group default qlen 1000
    link/ether 46:50:af:d9:5f:75 brd ff:ff:ff:ff:ff:ff
14: o-hm0@o-bhm0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether c2:c4:57:48:86:c0 brd ff:ff:ff:ff:ff:ff

sudo ip link set dev o-hm0 address $MGMT_PORT_MAC 実行後の状態

13: o-bhm0@o-hm0: <NO-CARRIER,BROADCAST,MULTICAST,UP,M-DOWN> mtu 1500 qdisc noqueue master brqa1859074-54 state LOWERLAYERDOWN mode DEFAULT group default qlen 1000
    link/ether 46:50:af:d9:5f:75 brd ff:ff:ff:ff:ff:ff
14: o-hm0@o-bhm0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether fa:16:3e:93:86:a7 brd ff:ff:ff:ff:ff:ff

oomichi commented 5 years ago

問題の dhclient を実行 → 問題なく動作した。アドレスを修正したため？

$ sudo dhclient -v o-hm0
Internet Systems Consortium DHCP Client 4.3.3
Copyright 2004-2015 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
Listening on LPF/o-hm0/fa:16:3e:93:86:a7
Sending on   LPF/o-hm0/fa:16:3e:93:86:a7
Sending on   Socket/fallback
DHCPREQUEST of 172.16.0.2 on o-hm0 to 255.255.255.255 port 67 (xid=0x3bfb38a9)
DHCPREQUEST of 172.16.0.2 on o-hm0 to 255.255.255.255 port 67 (xid=0x3bfb38a9)
DHCPDISCOVER on o-hm0 to 255.255.255.255 port 67 interval 3 (xid=0x9b84d97f)
DHCPREQUEST of 192.168.0.2 on o-hm0 to 255.255.255.255 port 67 (xid=0x7fd9849b)
DHCPOFFER of 192.168.0.2 from 192.168.0.100
DHCPACK of 192.168.0.2 from 192.168.0.100

bound to 192.168.0.2 -- renewal in 39731 seconds.
$

oomichi commented 5 years ago

Network として問題なくなった！

oomichi / try-kubernetes

Octavia: Connect provider network and lb-mgmt-net #91

環境のクリーンアップ

Octavia 公式（予定）の手順で構築する