search

zstackio / zstack

ZStack - the open-source IaaS software http://zstack.org (国内用户请至 http://zstack.io)
Apache License 2.0
1.3k stars 393 forks source link

zstack 云主机无法访问外部网络 #1305

Closed xuanyuanaosheng closed 2 years ago

xuanyuanaosheng commented 2 years ago

zstack 的架构，zstack目前只有一个节点，按照引导进行配置，使用的是扁平网络的网络模式。虚拟机可以正常开，虚拟机直接可以互相访问。但是无法访问公司其他网络。

Zstack架构 使用VLAN: 1935 物理节点的IP地址：10.26.246.18 云主机所在的IP地址：

物理机配置：

###  ifcfg-eno49 + ifcfg-eno50 组成 bond0, 使用 bond0.1935 子接口，桥接使用cloudbr0
# ls
ifcfg-eno49  ifdown-bnep  ifdown-isdn    ifdown-sit       ifup          ifup-ippp  ifup-plusb   ifup-sit       ifup-wireless
ifcfg-bond0       ifcfg-eno50  ifdown-eth   ifdown-post    ifdown-Team      ifup-aliases  ifup-ipv6  ifup-post    ifup-Team      init.ipv6-global
ifcfg-bond0.1935  ifcfg-lo     ifdown-ippp  ifdown-ppp     ifdown-TeamPort  ifup-bnep     ifup-isdn  ifup-ppp     ifup-TeamPort  network-functions
ifcfg-cloudbr0    ifdown       ifdown-ipv6

# ifconfig | grep inet
        inet6 fe80::9edc:71ff:fed0:18e0  prefixlen 64  scopeid 0x20<link>
        inet6 fe80::9edc:71ff:fed0:18e0  prefixlen 64  scopeid 0x20<link>
        inet6 fe80::2835:77ff:fed9:e18b  prefixlen 64  scopeid 0x20<link>
        inet 169.254.64.1  netmask 255.255.192.0  broadcast 0.0.0.0
        inet6 fe80::c070:14ff:fede:c7d6  prefixlen 64  scopeid 0x20<link>
        inet 10.26.246.18  netmask 255.255.255.0  broadcast 10.26.246.255
        inet6 fe80::9edc:71ff:fed0:18e0  prefixlen 64  scopeid 0x20<link>
        inet6 fe80::9edc:71ff:fed0:18e0  prefixlen 64  scopeid 0x20<link>
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        inet6 fe80::c8dc:41ff:feb4:a356  prefixlen 64  scopeid 0x20<link>
        inet6 fe80::60d4:97ff:fedd:1a99  prefixlen 64  scopeid 0x20<link>
        inet6 fe80::fcf8:d7ff:fea3:9500  prefixlen 64  scopeid 0x20<link>
        inet6 fe80::fcc2:b5ff:fe7e:a300  prefixlen 64  scopeid 0x20<link>
        inet6 fe80::fcfc:a1ff:fe8a:f300  prefixlen 64  scopeid 0x20<link>
        inet6 fe80::fcf2:bdff:fefc:4300  prefixlen 64  scopeid 0x20<link>
[root@whdczstack001 ~]# clear
[root@whdczstack001 ~]# ifconfig
bond0: flags=5187<UP,BROADCAST,RUNNING,MASTER,MULTICAST>  mtu 1500
        inet6 fe80::9edc:71ff:fed0:18e0  prefixlen 64  scopeid 0x20<link>
        ether 9c:dc:71:d0:18:e0  txqueuelen 1000  (Ethernet)
        RX packets 933380562  bytes 219592768414 (204.5 GiB)
        RX errors 0  dropped 1610995  overruns 0  frame 0
        TX packets 11523326  bytes 9848205886 (9.1 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

bond0.1935: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::9edc:71ff:fed0:18e0  prefixlen 64  scopeid 0x20<link>
        ether 9c:dc:71:d0:18:e0  txqueuelen 1000  (Ethernet)
        RX packets 7299082  bytes 13250614933 (12.3 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5627123  bytes 9367190243 (8.7 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

br_17df68a_1935: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::2835:77ff:fed9:e18b  prefixlen 64  scopeid 0x20<link>
        ether 9c:dc:71:d0:18:e0  txqueuelen 1000  (Ethernet)
        RX packets 463100  bytes 76255807 (72.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 474869  bytes 526154986 (501.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

br_conn_all_ns: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 65500
        inet 169.254.64.1  netmask 255.255.192.0  broadcast 0.0.0.0
        inet6 fe80::c070:14ff:fede:c7d6  prefixlen 64  scopeid 0x20<link>
        ether 62:d4:97:dd:1a:99  txqueuelen 1000  (Ethernet)
        RX packets 383766  bytes 525430022 (501.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 383766  bytes 35306486 (33.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

cloudbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.26.246.18  netmask 255.255.255.0  broadcast 10.26.246.255
        inet6 fe80::9edc:71ff:fed0:18e0  prefixlen 64  scopeid 0x20<link>
        ether 9c:dc:71:d0:18:e0  txqueuelen 1000  (Ethernet)
        RX packets 7299059  bytes 13250430005 (12.3 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5411053  bytes 9355910463 (8.7 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

cloudbr0.1935: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::9edc:71ff:fed0:18e0  prefixlen 64  scopeid 0x20<link>
        ether 9c:dc:71:d0:18:e0  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 10960  bytes 461392 (450.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eno49: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST>  mtu 1500
        ether 9c:dc:71:d0:18:e0  txqueuelen 1000  (Ethernet)
        RX packets 933380533  bytes 219592766558 (204.5 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 11523326  bytes 9848205886 (9.1 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 16  memory 0x94000000-947fffff  

eno50: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST>  mtu 1500
        ether 9c:dc:71:d0:18:e0  txqueuelen 1000  (Ethernet)
        RX packets 29  bytes 1856 (1.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 17  memory 0x93000000-937fffff  

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 25289413  bytes 12403720866 (11.5 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 25289413  bytes 12403720866 (11.5 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

outer0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 65500
        inet6 fe80::c8dc:41ff:feb4:a356  prefixlen 64  scopeid 0x20<link>
        ether ca:dc:41:b4:a3:56  txqueuelen 1000  (Ethernet)
        RX packets 463100  bytes 76255807 (72.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 474869  bytes 526154986 (501.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ud_outer0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 65500
        inet6 fe80::60d4:97ff:fedd:1a99  prefixlen 64  scopeid 0x20<link>
        ether 62:d4:97:dd:1a:99  txqueuelen 1000  (Ethernet)
        RX packets 383766  bytes 530802746 (506.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 383771  bytes 35307032 (33.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vnic4.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::fcf8:d7ff:fea3:9500  prefixlen 64  scopeid 0x20<link>
        ether fe:f8:d7:a3:95:00  txqueuelen 1000  (Ethernet)
        RX packets 258984  bytes 296128368 (282.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 275054  bytes 41450497 (39.5 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vnic5.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::fcc2:b5ff:fe7e:a300  prefixlen 64  scopeid 0x20<link>
        ether fe:c2:b5:7e:a3:00  txqueuelen 1000  (Ethernet)
        RX packets 207930  bytes 229689500 (219.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 227832  bytes 36999554 (35.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vnic6.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::fcfc:a1ff:fe8a:f300  prefixlen 64  scopeid 0x20<link>
        ether fe:fc:a1:8a:f3:00  txqueuelen 1000  (Ethernet)
        RX packets 3999  bytes 168810 (164.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 6264  bytes 266572 (260.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vnic7.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::fcf2:bdff:fefc:4300  prefixlen 64  scopeid 0x20<link>
        ether fe:f2:bd:fc:43:00  txqueuelen 1000  (Ethernet)
        RX packets 3858  bytes 162888 (159.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 6405  bytes 272494 (266.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

 ## 能 ping 通网关 
 # ping 10.26.246.254
PING 10.26.246.254 (10.26.246.254) 56(84) bytes of data.
64 bytes from 10.26.246.254: icmp_seq=1 ttl=255 time=0.463 ms
64 bytes from 10.26.246.254: icmp_seq=2 ttl=255 time=145 ms
^C
--- 10.26.246.254 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms

# route -n 
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.26.246.254   0.0.0.0         UG    0      0        0 cloudbr0
10.26.246.0     0.0.0.0         255.255.255.0   U     0      0        0 cloudbr0
169.254.0.0     0.0.0.0         255.255.0.0     U     1004   0        0 bond0
169.254.0.0     0.0.0.0         255.255.0.0     U     1006   0        0 cloudbr0
169.254.64.0    0.0.0.0         255.255.192.0   U     0      0        0 br_conn_all_ns

云主机状态：

  1. 云主机之间可以互通
  2. 云主机可以正常开出来
  3. 云主机可以控制台登陆 image

目前的情况是

  1. 云主机无法ping 通网关：10.26.246.254
  2. 外部无法ssh登陆云主机

请问这是什么原因？怎么解决这个问题？

xuanyuanaosheng commented 2 years ago

@MatheMatrix @AlanJager

MatheMatrix commented 2 years ago

在 host 上执行 brctl show 看下

xuanyuanaosheng commented 2 years ago

@MatheMatrix 

# brctl show
bridge name bridge id       STP enabled interfaces
br_17df68a_1935     8000.9cdc71d018e0   no      cloudbr0.1935
                            outer0
                            vnic4.0
                            vnic5.0
                            vnic6.0
                            vnic7.0
br_conn_all_ns      8000.62d497dd1a99   no      ud_outer0
cloudbr0        8000.9cdc71d018e0   yes     bond0.1935

以及二层网络的配置：

image

MatheMatrix commented 2 years ago

相当于 

eno49+ eno50 --> bond0 --> bond0.1935 --> cloudbr0 --> cloudbr0.1935 --> vnic

这样会封装两次 1935 的 vlan 标签

在云平台创建二层网络的时候直接用 bond0 + vlan 1935 或者 cloudbr0 + novlan 即可

xuanyuanaosheng commented 2 years ago

@MatheMatrix 经过测试

  1. 使用cloudbr0 + novlan 创建二层网络会失败，进而无法加载集群

image

具体的报错:

在物理机[uuid:9981ffa5c12842e5b658515f0572c766]上为L2网络[uuid:33b41e26d8124d35b4d04809419adec8, type:L2NoVlanNetwork]创建网桥[br_cloudbr0]失败，原因: unable to create bridge[br_cloudbr0] from device[cloudbr0], because interface cloudbr0 is bridge
  1. 使用bond0 + vlan 1935 创建二层网络也会失败，进而无法加载集群

image

具体的报错:

创建L2网络[uuid:42cb10ce04e84309980e5319d3b5dbdd中的网桥[br_bond0_1935]失败 , 类型为: L2VlanNetwork, vlan:1,935] 在物理机[uuid:9981ffa5c12842e5b658515f0572c766]上, 原因: unable to create vlan bridge[name:br_bond0_1935, vlan:1935] from device[bond0], because failed to create bridge[br_bond0_1935], physical interface[bond0.1935] has been occupied by bridge[cloudbr0]

请问是我哪里配置的不对么？

反而使用：cloudbr0 + vlan 1935 能创建成功

image

MatheMatrix commented 2 years ago

把 cloudbr0 删掉，然后用 bond0 + vlan 1935 创建

可以参考 https://www.zstack.io/help/tutorials/flat_tutorial/v4/1.html

xuanyuanaosheng commented 2 years ago

@MatheMatrix 好的，我测试下看看。 去掉 cloudbr0 后测试通过。

具体配置： eno49+ eno50 --> bond0 --> bond0.1935 --> vnic

# brctl show
bridge name bridge id       STP enabled interfaces
br_bond0_1935       8000.9cdc71d018e0   no      bond0.1935
                            outer0
                            vnic10.0
                            vnic8.0
                            vnic9.0
br_conn_all_ns      8000.3a3e27d7e777   no      ud_outer0

  1. ifcfg-eno49 网卡配置

    TYPE="Ethernet"
BOOTPROTO="none"
NAME="bond0-slave"
DEVICE="eno49"
MASTER=bond0
SLAVE=yes
ONBOOT=yes
HOTPLUG=no
DELAY=0
STP=no

  2. ifcfg-eno50 网卡配置

    
TYPE=Ethernet
BOOTPROTO=none
NAME=bond0-slave
MASTER=bond0
SLAVE=yes
DEVICE=eno50
ONBOOT=yes
HOTPLUG=no
DELAY=0
STP=no

3. ifcfg-bond0 的配置

NAME=bond0 DEVICE=bond0 TYPE=Bond BOOTPROTO=none BONDING_MASTER=yes ONBOOT=yes HOTPLUG=no BONDING_OPTS="mode=1 miimon=100"


4. ifcfg-bond0.1935

DEVICE=bond0.1935 NAME=bond0.1935 BOOTPROTO=none ONBOOT=yes HOTPLUG=no VLAN=yes IPADDR=10.26.246.33 GATEWAY=10.26.246.254 NETMASK=255.255.255.0



Zstack 二层网络的配置：

![image](https://user-images.githubusercontent.com/4197714/177257449-3ebfa804-d5eb-4fd3-a4db-a89a20e08c8f.png)

Zstack 三层网络的配置：

![image](https://user-images.githubusercontent.com/4197714/177257529-f3bb36ba-294d-462e-afbc-4d8419b53e16.png)