Closed oomichi closed 5 years ago
Routerの削除 → 失敗
$ openstack router delete router01
Failed to delete router with name or ID 'router01': Unable to delete Router for openstack.network.v2.router.Router(status=ACTIVE, external_gateway_info={u'network_id': u'bfd9fd43-c9b4-43ad-bb67-930c674f2605', u'enable_snat': True, u'external_fixed_ips': [{u'subnet_id': u'43ed897b-3c10-4d5c-8f6d-263edcd817c7', u'ip_address': u'192.168.1.115'}]}, availability_zone_hints=[], availability_zones=[], description=, tags=[], tenant_id=682e74f275fe427abd9eb6759f3b68c5, created_at=2019-05-07T02:22:48Z, admin_state_up=True, distributed=False, updated_at=2019-05-07T02:25:43Z, flavor_id=None, routes=[], ha=False, revision=4, id=8f079f83-1585-4783-b188-2f62141c1b5c, name=router01)
1 of 1 routers failed to delete.
Router から subnet を削除 → Router削除が成功
$ openstack router remove subnet router01 lb-mgmt-subnet
$ openstack router remove subnet router01 provider
$ openstack router delete router01
Octavia管理用サブネットを削除
$ openstack subnet delete lb-mgmt-subnet
https://review.opendev.org/#/c/672842/ の手順
$ OCTAVIA_MGMT_SUBNET=172.16.0.0/12
$ OCTAVIA_MGMT_SUBNET_START=172.16.0.100
$ OCTAVIA_MGMT_SUBNET_END=172.16.31.254
$ OCTAVIA_MGMT_PORT_IP=172.16.0.2
$ openstack network create lb-mgmt-net
$ openstack subnet create --subnet-range $OCTAVIA_MGMT_SUBNET \
--allocation-pool start=$OCTAVIA_MGMT_SUBNET_START,\
end=$OCTAVIA_MGMT_SUBNET_END --network lb-mgmt-net lb-mgmt-subnet
$ SUBNET_ID=$(openstack subnet show lb-mgmt-subnet -f value -c id)
$ PORT_FIXED_IP="--fixed-ip subnet=$SUBNET_ID,ip-address=$OCTAVIA_MGMT_PORT_IP"
$ MGMT_PORT_ID=$(openstack port create --security-group \
lb-health-mgr-sec-grp --device-owner Octavia:health-mgr \
--host=$(hostname) -c id -f value --network lb-mgmt-net \
$PORT_FIXED_IP octavia-health-manager-listen-port)
$ MGMT_PORT_MAC=$(openstack port show -c mac_address -f value \
$MGMT_PORT_ID)
$ MGMT_PORT_IP=$(openstack port show -f value -c fixed_ips \
$MGMT_PORT_ID | awk '{FS=",| "; gsub(",",""); gsub("'\''",""); \
for(i = 1; i <= NF; ++i) {if ($i ~ /^ip_address/) {n=index($i, "="); \
if (substr($i, n+1) ~ "\\.") print substr($i, n+1)}}}')
$ sudo ip link add o-hm0 type veth peer name o-bhm0
$ NETID=$(openstack network show lb-mgmt-net -c id -f value)
$ BRNAME=brq$(echo $NETID|cut -c 1-11)
$ sudo brctl addif $BRNAME o-bhm0
$ sudo ip link set o-bhm0 up
$ sudo ip link set dev o-hm0 address $MGMT_PORT_MAC
$ sudo iptables -I INPUT -i o-hm0 -p udp --dport 5555 -j ACCEPT
$ sudo dhclient -v o-hm0
最後の sudo dhclient -v o-hm0
を実行したタイミングでネットワークが不通になる。
devstack で指定している下記の内容の設定ファイルを指定しても同様。
request subnet-mask,broadcast-address,interface-mtu;
do-forward-updates false;
下記のように api ノードではない場合に実行することになっている
500 function octavia_start {
501
502 if ! ps aux | grep -q [o]-hm0 && [ $OCTAVIA_NODE != 'api' ] ; then
503 sudo dhclient -v o-hm0 -cf $OCTAVIA_DHCLIENT_CONF
504 fi
docker0 のアドレスが 172.17.0.1 となっている。 このネットワークアドレスは docker のデフォルト。
docker0 Link encap:Ethernet HWaddr 02:42:66:9c:c4:5e
inet addr:172.17.0.1 Bcast:172.17.255.255 Mask:255.255.0.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
これが 172.16.0.0/12 の lb-mgmt-subnet のアドレスと重複している。 重複しないように変更したほうが良さそう。 devstack での設定値を調べてみる。 octavia/devstack/settings
OCTAVIA_MGMT_SUBNET=${OCTAVIA_MGMT_SUBNET:-"192.168.0.0/24"}
OCTAVIA_MGMT_SUBNET_START=${OCTAVIA_MGMT_SUBNET_START:-"192.168.0.2"}
OCTAVIA_MGMT_SUBNET_END=${OCTAVIA_MGMT_SUBNET_END:-"192.168.0.200"}
192.168.1.0/24は使っているが、192.168.0.0/24 は使っていないのでこれを利用する。
$ OCTAVIA_MGMT_SUBNET=192.168.0.0/24
$ OCTAVIA_MGMT_SUBNET_START=192.168.0.100
$ OCTAVIA_MGMT_SUBNET_END=192.168.0.200
$ OCTAVIA_MGMT_PORT_IP=192.168.0.2
$ openstack network create lb-mgmt-net
$ openstack subnet create --subnet-range $OCTAVIA_MGMT_SUBNET \
--allocation-pool start=$OCTAVIA_MGMT_SUBNET_START,end=$OCTAVIA_MGMT_SUBNET_END \
--network lb-mgmt-net lb-mgmt-subnet
$ SUBNET_ID=$(openstack subnet show lb-mgmt-subnet -f value -c id)
$ PORT_FIXED_IP="--fixed-ip subnet=$SUBNET_ID,ip-address=$OCTAVIA_MGMT_PORT_IP"
$ MGMT_PORT_ID=$(openstack port create --security-group \
lb-health-mgr-sec-grp --device-owner Octavia:health-mgr \
--host=$(hostname) -c id -f value --network lb-mgmt-net \
$PORT_FIXED_IP octavia-health-manager-listen-port)
$ MGMT_PORT_MAC=$(openstack port show -c mac_address -f value $MGMT_PORT_ID)
$ MGMT_PORT_IP=$(openstack port show -f value -c fixed_ips \
$MGMT_PORT_ID | awk '{FS=",| "; gsub(",",""); gsub("'\''",""); \
for(i = 1; i <= NF; ++i) {if ($i ~ /^ip_address/) {n=index($i, "="); \
if (substr($i, n+1) ~ "\\.") print substr($i, n+1)}}}')
$ sudo ip link add o-hm0 type veth peer name o-bhm0
$ NETID=$(openstack network show lb-mgmt-net -c id -f value)
$ BRNAME=brq$(echo $NETID|cut -c 1-11)
$ sudo brctl addif $BRNAME o-bhm0
$ sudo ip link set o-bhm0 up
$ sudo ip link set dev o-hm0 address $MGMT_PORT_MAC
$ sudo iptables -I INPUT -i o-hm0 -p udp --dport 5555 -j ACCEPT
$ sudo dhclient -v o-hm0
sudo ip link set o-bhm0 up
実行直前の状態
13: o-bhm0@o-hm0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop master brqa1859074-54 state DOWN mode DEFAULT group default qlen 1000
link/ether 46:50:af:d9:5f:75 brd ff:ff:ff:ff:ff:ff
14: o-hm0@o-bhm0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether c2:c4:57:48:86:c0 brd ff:ff:ff:ff:ff:ff
sudo ip link set o-bhm0 up
実行後の状態
13: o-bhm0@o-hm0: <NO-CARRIER,BROADCAST,MULTICAST,UP,M-DOWN> mtu 1500 qdisc noqueue master brqa1859074-54 state LOWERLAYERDOWN mode DEFAULT group default qlen 1000
link/ether 46:50:af:d9:5f:75 brd ff:ff:ff:ff:ff:ff
14: o-hm0@o-bhm0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether c2:c4:57:48:86:c0 brd ff:ff:ff:ff:ff:ff
sudo ip link set dev o-hm0 address $MGMT_PORT_MAC
実行後の状態
13: o-bhm0@o-hm0: <NO-CARRIER,BROADCAST,MULTICAST,UP,M-DOWN> mtu 1500 qdisc noqueue master brqa1859074-54 state LOWERLAYERDOWN mode DEFAULT group default qlen 1000
link/ether 46:50:af:d9:5f:75 brd ff:ff:ff:ff:ff:ff
14: o-hm0@o-bhm0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether fa:16:3e:93:86:a7 brd ff:ff:ff:ff:ff:ff
問題の dhclient を実行 → 問題なく動作した。 アドレスを修正したため?
$ sudo dhclient -v o-hm0
Internet Systems Consortium DHCP Client 4.3.3
Copyright 2004-2015 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
Listening on LPF/o-hm0/fa:16:3e:93:86:a7
Sending on LPF/o-hm0/fa:16:3e:93:86:a7
Sending on Socket/fallback
DHCPREQUEST of 172.16.0.2 on o-hm0 to 255.255.255.255 port 67 (xid=0x3bfb38a9)
DHCPREQUEST of 172.16.0.2 on o-hm0 to 255.255.255.255 port 67 (xid=0x3bfb38a9)
DHCPDISCOVER on o-hm0 to 255.255.255.255 port 67 interval 3 (xid=0x9b84d97f)
DHCPREQUEST of 192.168.0.2 on o-hm0 to 255.255.255.255 port 67 (xid=0x7fd9849b)
DHCPOFFER of 192.168.0.2 from 192.168.0.100
DHCPACK of 192.168.0.2 from 192.168.0.100
bound to 192.168.0.2 -- renewal in 39731 seconds.
$
Network として問題なくなった!
外向けのネットワーク provider と Octavia 管理用ネットワーク lb-mgmt-net を疎通させる。
https://github.com/oomichi/try-kubernetes/issues/68#issuecomment-528601993