ISSUE TYPE

Other

COMPONENT NAME

Secondary Storage VM

CLOUDSTACK VERSION

4.17

CONFIGURATION

advanced networking.

OS / ENVIRONMENT

ubuntu 22.04.1 LTS

SUMMARY

I got a running secondary storage vm. But it's not reachable to upload an image to it's zone.

STEPS TO REPRODUCE

Set up the advanced network management server, add a zone + host. Add Secondary Storage. Try to upload an ISO image to the zone.

ACTUAL RESULTS

UI throws: Request failed. (530) There is no secondary storage VM for downloading template to image store bla Running /usr/local/cloud/systemvm/ssvm-check.sh inside the ssvm thows: WARNING: cannot ping DNS server and ERROR: DNS not resolving cloudstack.apache.org Pinging the gateway works.

The management server logs say there is no SSVM: management-server.log

@MilanHofmann there must be information before what you shared in the management server log. Also, can you check if you can resolve cloudstack.apache.org using the internal and external DNS you used when creating the zone?

@DaanHoogland Thank you, for the quick answer!

here are the logs with more events before the error: management-server.log I checked the firewall rules and now it resolves!

Now the check gives the following output:

First DNS server is 8.8.8.8 PING 8.8.8.8 (8.8.8.8): 56 data bytes 64 bytes from 8.8.8.8: icmp_seq=0 ttl=58 time=11.732 ms 64 bytes from 8.8.8.8: icmp_seq=1 ttl=58 time=11.725 ms --- 8.8.8.8 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max/stddev = 11.725/11.729/11.732/0.000 ms Good: Can ping DNS server

Good: DNS resolves cloudstack.apache.org

ERROR: NFS is not currently mounted Try manually mounting from inside the VM NFS server is 255.255.255.0 PING 255.255.255.0 (255.255.255.0): 56 data bytes 92 bytes from s-12-VM (79.143.179.254): Destination Host Unreachable 92 bytes from s-12-VM (79.143.179.254): Destination Host Unreachable --- 255.255.255.0 ping statistics --- 2 packets transmitted, 0 packets received, 100% packet loss WARNING: cannot ping nfs server routing table follows Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 79.143.179.1 0.0.0.0 UG 0 0 0 eth2 8.8.8.8 79.143.179.1 255.255.255.255 UGH 0 0 0 eth1 10.0.0.0 79.143.179.1 255.0.0.0 UG 0 0 0 eth1 79.143.179.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1 79.143.179.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 172.16.0.0 79.143.179.1 255.240.0.0 UG 0 0 0 eth1 172.31.16.0 79.143.179.1 255.255.240.0 UG 0 0 0 eth1 172.31.28.117 79.143.179.1 255.255.255.255 UGH 0 0 0 eth1 192.168.0.0 79.143.179.1 255.255.0.0 UG 0 0 0 eth1

Management server is 172.31.28.117. Checking connectivity. 2023/02/09 20:01:11 socat[3620] E connecting to AF=2 172.31.28.117:8250: Connection timed out Good: Can connect to management server 172.31.28.117 port 8250

Good: Java process is running ================================================_

Is it correct that the nfs is routed to 255.255.255.0? And it seems, that the management server sent his private ip (172.31.28.117) instead of the public one to the host. Is there a way to configure this right on the management server?

@MilanHofmann You can update the global setting "host" to the public ip of mgmt server.

@weizhouapache Thank you! Still the nfs server is not reachable ->

First DNS server is 8.8.8.8 PING 8.8.8.8 (8.8.8.8): 56 data bytes --- 8.8.8.8 ping statistics --- 2 packets transmitted, 0 packets received, 100% packet loss WARNING: cannot ping DNS server route follows Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 8.8.8.8 79.143.179.1 255.255.255.255 UGH 0 0 0 eth1 10.0.0.0 79.143.179.1 255.0.0.0 UG 0 0 0 eth1 79.143.179.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 172.16.0.0 79.143.179.1 255.240.0.0 UG 0 0 0 eth1 172.31.16.0 79.143.179.1 255.255.240.0 UG 0 0 0 eth1 192.168.0.0 79.143.179.1 255.255.0.0 UG 0 0 0 eth1

Good: DNS resolves cloudstack.apache.org

ERROR: NFS is not currently mounted Try manually mounting from inside the VM NFS server is 255.255.255.0 PING 255.255.255.0 (255.255.255.0): 56 data bytes ping: sending packet: Network is unreachable WARNING: cannot ping nfs server routing table follows Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 8.8.8.8 79.143.179.1 255.255.255.255 UGH 0 0 0 eth1 10.0.0.0 79.143.179.1 255.0.0.0 UG 0 0 0 eth1 79.143.179.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 172.16.0.0 79.143.179.1 255.240.0.0 UG 0 0 0 eth1 172.31.16.0 79.143.179.1 255.255.240.0 UG 0 0 0 eth1 192.168.0.0 79.143.179.1 255.255.0.0 UG 0 0 0 eth1

Management server is $externalIP. Checking connectivity. 2023/02/10 06:30:16 socat[3407] E connect(5, AF=2 $externalIP:8250, 16): Network is unreachable Good: Can connect to management server $externalIP port 8250

Good: Java process is running

In the console, the systemvm stays in Agent state "connecting" then changes to "alert". Console proxy is up and running. But both cannot connect when trying to open the console via the ui.

@weizhouapache Thank you! Still the nfs server is not reachable ->

First DNS server is 8.8.8.8

PING 8.8.8.8 (8.8.8.8): 56 data bytes --- 8.8.8.8 ping statistics --- 2 packets transmitted, 0 packets received, 100% packet loss WARNING: cannot ping DNS server route follows Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 8.8.8.8 79.143.179.1 255.255.255.255 UGH 0 0 0 eth1 10.0.0.0 79.143.179.1 255.0.0.0 UG 0 0 0 eth1 79.143.179.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 172.16.0.0 79.143.179.1 255.240.0.0 UG 0 0 0 eth1 172.31.16.0 79.143.179.1 255.255.240.0 UG 0 0 0 eth1 192.168.0.0 79.143.179.1 255.255.0.0 UG 0 0 0 eth1

Good: DNS resolves cloudstack.apache.org

ERROR: NFS is not currently mounted

Try manually mounting from inside the VM NFS server is 255.255.255.0 PING 255.255.255.0 (255.255.255.0): 56 data bytes ping: sending packet: Network is unreachable WARNING: cannot ping nfs server routing table follows Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 8.8.8.8 79.143.179.1 255.255.255.255 UGH 0 0 0 eth1 10.0.0.0 79.143.179.1 255.0.0.0 UG 0 0 0 eth1 79.143.179.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 172.16.0.0 79.143.179.1 255.240.0.0 UG 0 0 0 eth1 172.31.16.0 79.143.179.1 255.255.240.0 UG 0 0 0 eth1 192.168.0.0 79.143.179.1 255.255.0.0 UG 0 0 0 eth1

Management server is $externalIP. Checking connectivity.

2023/02/10 06:30:16 socat[3407] E connect(5, AF=2 $externalIP:8250, 16): Network is unreachable Good: Can connect to management server $externalIP port 8250

Good: Java process is running

In the console, the systemvm stays in Agent state "connecting" then changes to "alert". Console proxy is up and running. But both cannot connect when trying to open the console via the ui.

@MilanHofmann SSVM does not have a route to the managament server IP. It does not have a default route actually

@weizhouapache Thank you, that was the problem!

If I add the default route manually the vm checks all pass and NFS can be mounted. But why is it necessary to add it myself? Shouldn't that be done by KVM during the launch?

Also the vm is still not reachable from outside the host. Maybe I misconfigured something here? root@m0995:~# ip a gives 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master cloudbr0 state UP group default qlen 1000 link/ether 1c:6f:65:91:9e:86 brd ff:ff:ff:ff:ff:ff altname enp4s0 4: cloudbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 1c:6f:65:91:9e:86 brd ff:ff:ff:ff:ff:ff inet 79.143.179.221/24 metric 100 brd 79.143.179.255 scope global dynamic cloudbr0 valid_lft 3777sec preferred_lft 3777sec inet6 fe80::1e6f:65ff:fe91:9e86/64 scope link valid_lft forever preferred_lft forever 5: cloudbr1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000 link/ether 1c:6f:65:91:9e:86 brd ff:ff:ff:ff:ff:ff inet6 fe80::1e6f:65ff:fe91:9e86/64 scope link valid_lft forever preferred_lft forever 7: cloud0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether a6:d6:15:d8:3a:12 brd ff:ff:ff:ff:ff:ff inet 169.254.0.1/16 scope global cloud0 valid_lft forever preferred_lft forever 40: eth0.200@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master breth0-200 state UP group default qlen 1000 link/ether 1c:6f:65:91:9e:86 brd ff:ff:ff:ff:ff:ff 41: breth0-200: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 82:8d:82:b1:a4:f2 brd ff:ff:ff:ff:ff:ff 42: vnet18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cloud0 state UNKNOWN group default qlen 1000 link/ether fe:00:a9:fe:b9:23 brd ff:ff:ff:ff:ff:ff 43: vnet19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cloudbr0 state UNKNOWN group default qlen 1000 link/ether fe:00:e3:00:01:7b brd ff:ff:ff:ff:ff:ff 44: vnet20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master breth0-200 state UNKNOWN group default qlen 1000 link/ether fe:00:14:00:00:69 brd ff:ff:ff:ff:ff:ff 45: vnet21: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cloud0 state UNKNOWN group default qlen 1000 link/ether fe:00:a9:fe:a1:f9 brd ff:ff:ff:ff:ff:ff 46: vnet22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cloudbr0 state UNKNOWN group default qlen 1000 link/ether fe:00:7d:00:01:1c brd ff:ff:ff:ff:ff:ff 47: vnet23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master breth0-200 state UNKNOWN group default qlen 1000 link/ether fe:00:ed:00:00:68 brd ff:ff:ff:ff:ff:ff 48: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000 link/ether 52:54:00:42:23:f0 brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 valid_lft forever preferred_lft forever

this is my netplan config:

network: version: 2 renderer: networkd ethernets: ens18: match: macaddress: $myMacAdress bridges: cloudbr0: dhcp4: true macaddress: $myMacAdress interfaces:

ens18 addresses: [$myAddress]
gateway4 is deprecated, use routes instead

routes:
to: 0.0.0.0/0 via: $myGateway on-link: true nameservers: search: [ invalid ] addresses:
- 8.8.8.8
- 8.8.4.4 cloudbr1: dhcp4: false macaddress: $myMacAdress optional: true

@MilanHofmann

can you describe your configuration ? mgmt server / agent ,nics/ ips, etc

@weizhouapache Thank you, that was the problem!

If I add the default route manually the vm checks all pass and NFS can be mounted. But why is it necessary to add it myself? Shouldn't that be done by KVM during the launch?

Also the vm is still not reachable from outside the host. Maybe I misconfigured something here? root@m0995:~# ip a gives 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master cloudbr0 state UP group default qlen 1000 link/ether 1c:6f:65:91:9e:86 brd ff:ff:ff:ff:ff:ff altname enp4s0 4: cloudbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 1c:6f:65:91:9e:86 brd ff:ff:ff:ff:ff:ff inet 79.143.179.221/24 metric 100 brd 79.143.179.255 scope global dynamic cloudbr0 valid_lft 3777sec preferred_lft 3777sec inet6 fe80::1e6f:65ff:fe91:9e86/64 scope link valid_lft forever preferred_lft forever 5: cloudbr1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000 link/ether 1c:6f:65:91:9e:86 brd ff:ff:ff:ff:ff:ff inet6 fe80::1e6f:65ff:fe91:9e86/64 scope link valid_lft forever preferred_lft forever 7: cloud0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether a6:d6:15:d8:3a:12 brd ff:ff:ff:ff:ff:ff inet 169.254.0.1/16 scope global cloud0 valid_lft forever preferred_lft forever 40: eth0.200@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master breth0-200 state UP group default qlen 1000 link/ether 1c:6f:65:91:9e:86 brd ff:ff:ff:ff:ff:ff 41: breth0-200: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 82:8d:82:b1:a4:f2 brd ff:ff:ff:ff:ff:ff 42: vnet18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cloud0 state UNKNOWN group default qlen 1000 link/ether fe:00:a9:fe:b9:23 brd ff:ff:ff:ff:ff:ff 43: vnet19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cloudbr0 state UNKNOWN group default qlen 1000 link/ether fe:00:e3:00:01:7b brd ff:ff:ff:ff:ff:ff 44: vnet20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master breth0-200 state UNKNOWN group default qlen 1000 link/ether fe:00:14:00:00:69 brd ff:ff:ff:ff:ff:ff 45: vnet21: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cloud0 state UNKNOWN group default qlen 1000 link/ether fe:00:a9:fe:a1:f9 brd ff:ff:ff:ff:ff:ff 46: vnet22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cloudbr0 state UNKNOWN group default qlen 1000 link/ether fe:00:7d:00:01:1c brd ff:ff:ff:ff:ff:ff 47: vnet23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master breth0-200 state UNKNOWN group default qlen 1000 link/ether fe:00:ed:00:00:68 brd ff:ff:ff:ff:ff:ff 48: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000 link/ether 52:54:00:42:23:f0 brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 valid_lft forever preferred_lft forever

this is my netplan config:

network: version: 2 renderer: networkd ethernets: ens18: match: macaddress: $myMacAdress bridges: cloudbr0: dhcp4: true macaddress: $myMacAdress interfaces: - ens18 addresses: [$myAddress] # gateway4 is deprecated, use routes instead routes: - to: 0.0.0.0/0 via: $myGateway on-link: true nameservers: search: [ invalid ] addresses: - 8.8.8.8 - 8.8.4.4 cloudbr1: dhcp4: false macaddress: $myMacAdress optional: true

@MilanHofmann

can you describe your configuration ? mgmt server / agent ,nics/ ips, etc

@weizhouapache Thank you, that was the problem! If I add the default route manually the vm checks all pass and NFS can be mounted. But why is it necessary to add it myself? Shouldn't that be done by KVM during the launch? Also the vm is still not reachable from outside the host. Maybe I misconfigured something here? root@m0995:~# ip a gives 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master cloudbr0 state UP group default qlen 1000 link/ether 1c:6f:65:91:9e:86 brd ff:ff:ff:ff:ff:ff altname enp4s0 4: cloudbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 1c:6f:65:91:9e:86 brd ff:ff:ff:ff:ff:ff inet 79.143.179.221/24 metric 100 brd 79.143.179.255 scope global dynamic cloudbr0 valid_lft 3777sec preferred_lft 3777sec inet6 fe80::1e6f:65ff:fe91:9e86/64 scope link valid_lft forever preferred_lft forever 5: cloudbr1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000 link/ether 1c:6f:65:91:9e:86 brd ff:ff:ff:ff:ff:ff inet6 fe80::1e6f:65ff:fe91:9e86/64 scope link valid_lft forever preferred_lft forever 7: cloud0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether a6:d6:15:d8:3a:12 brd ff:ff:ff:ff:ff:ff inet 169.254.0.1/16 scope global cloud0 valid_lft forever preferred_lft forever 40: eth0.200@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master breth0-200 state UP group default qlen 1000 link/ether 1c:6f:65:91:9e:86 brd ff:ff:ff:ff:ff:ff 41: breth0-200: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 82:8d:82:b1:a4:f2 brd ff:ff:ff:ff:ff:ff 42: vnet18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cloud0 state UNKNOWN group default qlen 1000 link/ether fe:00:a9:fe:b9:23 brd ff:ff:ff:ff:ff:ff 43: vnet19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cloudbr0 state UNKNOWN group default qlen 1000 link/ether fe:00:e3:00:01:7b brd ff:ff:ff:ff:ff:ff 44: vnet20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master breth0-200 state UNKNOWN group default qlen 1000 link/ether fe:00:14:00:00:69 brd ff:ff:ff:ff:ff:ff 45: vnet21: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cloud0 state UNKNOWN group default qlen 1000 link/ether fe:00:a9:fe:a1:f9 brd ff:ff:ff:ff:ff:ff 46: vnet22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cloudbr0 state UNKNOWN group default qlen 1000 link/ether fe:00:7d:00:01:1c brd ff:ff:ff:ff:ff:ff 47: vnet23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master breth0-200 state UNKNOWN group default qlen 1000 link/ether fe:00:ed:00:00:68 brd ff:ff:ff:ff:ff:ff 48: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000 link/ether 52:54:00:42:23:f0 brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 valid_lft forever preferred_lft forever

this is my netplan config:

network: version: 2 renderer: networkd ethernets: ens18: match: macaddress: $myMacAdress bridges: cloudbr0: dhcp4: true macaddress: $myMacAdress interfaces: - ens18 addresses: [$myAddress] # gateway4 is deprecated, use routes instead routes: - to: 0.0.0.0/0 via: $myGateway on-link: true nameservers: search: [ invalid ] addresses: - 8.8.8.8 - 8.8.4.4 cloudbr1: dhcp4: false macaddress: $myMacAdress optional: true

@weizhouapache Sure!

ip route show on host gives default via 79.143.179.1 dev eth1 8.8.8.8 via 79.143.179.1 dev eth1 10.0.0.0/8 via 79.143.179.1 dev eth1 79.143.179.0/24 dev eth1 proto kernel scope link src 79.143.179.126 169.254.0.0/16 dev eth0 proto kernel scope link src 169.254.185.35 172.16.0.0/12 via 79.143.179.1 dev eth1 172.31.16.0/20 via 79.143.179.1 dev eth1 192.168.0.0/16 via 79.143.179.1 dev eth1 sudo iptables -S gives:

-P INPUT ACCEPT -P FORWARD ACCEPT -P OUTPUT ACCEPT -N BF-breth0-200 -N BF-breth0-200-IN -N BF-breth0-200-OUT -N BF-cloudbr0 -N BF-cloudbr0-IN -N BF-cloudbr0-OUT -N LIBVIRT_FWI -N LIBVIRT_FWO -N LIBVIRT_FWX -N LIBVIRT_INP -N LIBVIRT_OUT -N s-10-VM -N s-12-VM -N s-14-VM -N s-16-VM -N s-17-VM -N ufw-after-forward -N ufw-after-input -N ufw-after-logging-forward -N ufw-after-logging-input -N ufw-after-logging-output -N ufw-after-output -N ufw-before-forward -N ufw-before-input -N ufw-before-logging-forward -N ufw-before-logging-input -N ufw-before-logging-output -N ufw-before-output -N ufw-reject-forward -N ufw-reject-input -N ufw-reject-output -N ufw-track-forward -N ufw-track-input -N ufw-track-output -N v-11-VM -N v-13-VM -N v-15-VM -N v-18-VM -N v-9-VM -A INPUT -j LIBVIRT_INP -A INPUT -j ufw-before-logging-input -A INPUT -j ufw-before-input -A INPUT -j ufw-after-input -A INPUT -j ufw-after-logging-input -A INPUT -j ufw-reject-input -A INPUT -j ufw-track-input -A FORWARD -o cloudbr0 -m physdev --physdev-is-bridged -j BF-cloudbr0 -A FORWARD -i cloudbr0 -m physdev --physdev-is-bridged -j BF-cloudbr0 -A FORWARD -o cloudbr0 -j DROP -A FORWARD -i cloudbr0 -j DROP -A FORWARD -o breth0-200 -m physdev --physdev-is-bridged -j BF-breth0-200 -A FORWARD -i breth0-200 -m physdev --physdev-is-bridged -j BF-breth0-200 -A FORWARD -o breth0-200 -j DROP -A FORWARD -i breth0-200 -j DROP -A FORWARD -j LIBVIRT_FWX -A FORWARD -j LIBVIRT_FWI -A FORWARD -j LIBVIRT_FWO -A FORWARD -j ufw-before-logging-forward -A FORWARD -j ufw-before-forward -A FORWARD -j ufw-after-forward -A FORWARD -j ufw-after-logging-forward -A FORWARD -j ufw-reject-forward -A FORWARD -j ufw-track-forward -A OUTPUT -j LIBVIRT_OUT -A OUTPUT -j ufw-before-logging-output -A OUTPUT -j ufw-before-output -A OUTPUT -j ufw-after-output -A OUTPUT -j ufw-after-logging-output -A OUTPUT -j ufw-reject-output -A OUTPUT -j ufw-track-output -A BF-breth0-200 -m state --state RELATED,ESTABLISHED -j ACCEPT -A BF-breth0-200 -m physdev --physdev-is-in --physdev-is-bridged -j BF-breth0-200-IN -A BF-breth0-200 -m physdev --physdev-is-out --physdev-is-bridged -j BF-breth0-200-OUT -A BF-breth0-200 -m physdev --physdev-out eth0.200 --physdev-is-bridged -j ACCEPT -A BF-breth0-200-IN -m physdev --physdev-in vnet20 --physdev-is-bridged -j s-17-VM -A BF-breth0-200-IN -m physdev --physdev-in vnet23 --physdev-is-bridged -j v-18-VM -A BF-breth0-200-OUT -m physdev --physdev-out vnet20 --physdev-is-bridged -j s-17-VM -A BF-breth0-200-OUT -m physdev --physdev-out vnet23 --physdev-is-bridged -j v-18-VM -A BF-cloudbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT -A BF-cloudbr0 -m physdev --physdev-is-in --physdev-is-bridged -j BF-cloudbr0-IN -A BF-cloudbr0 -m physdev --physdev-is-out --physdev-is-bridged -j BF-cloudbr0-OUT -A BF-cloudbr0 -m physdev --physdev-out eth0 --physdev-is-bridged -j ACCEPT -A BF-cloudbr0-IN -m physdev --physdev-in vnet19 --physdev-is-bridged -j s-17-VM -A BF-cloudbr0-IN -m physdev --physdev-in vnet22 --physdev-is-bridged -j v-18-VM -A BF-cloudbr0-OUT -m physdev --physdev-out vnet19 --physdev-is-bridged -j s-17-VM -A BF-cloudbr0-OUT -m physdev --physdev-out vnet22 --physdev-is-bridged -j v-18-VM -A s-10-VM -m physdev --physdev-in vnet5 --physdev-is-bridged -j RETURN -A s-10-VM -m physdev --physdev-in vnet4 --physdev-is-bridged -j RETURN -A s-10-VM -j ACCEPT -A s-12-VM -m physdev --physdev-in vnet5 --physdev-is-bridged -j RETURN -A s-12-VM -m physdev --physdev-in vnet4 --physdev-is-bridged -j RETURN -A s-12-VM -j ACCEPT -A s-14-VM -m physdev --physdev-in vnet10 --physdev-is-bridged -j RETURN -A s-14-VM -m physdev --physdev-in vnet11 --physdev-is-bridged -j RETURN -A s-14-VM -j ACCEPT -A s-16-VM -m physdev --physdev-in vnet17 --physdev-is-bridged -j RETURN -A s-16-VM -m physdev --physdev-in vnet16 --physdev-is-bridged -j RETURN -A s-16-VM -j ACCEPT -A s-17-VM -m physdev --physdev-in vnet20 --physdev-is-bridged -j RETURN -A s-17-VM -m physdev --physdev-in vnet19 --physdev-is-bridged -j RETURN -A s-17-VM -j ACCEPT -A v-11-VM -m physdev --physdev-in vnet2 --physdev-is-bridged -j RETURN -A v-11-VM -m physdev --physdev-in vnet1 --physdev-is-bridged -j RETURN -A v-11-VM -j ACCEPT -A v-13-VM -m physdev --physdev-in vnet7 --physdev-is-bridged -j RETURN -A v-13-VM -m physdev --physdev-in vnet8 --physdev-is-bridged -j RETURN -A v-13-VM -j ACCEPT -A v-15-VM -m physdev --physdev-in vnet14 --physdev-is-bridged -j RETURN -A v-15-VM -m physdev --physdev-in vnet13 --physdev-is-bridged -j RETURN -A v-15-VM -j ACCEPT -A v-18-VM -m physdev --physdev-in vnet22 --physdev-is-bridged -j RETURN -A v-18-VM -m physdev --physdev-in vnet23 --physdev-is-bridged -j RETURN -A v-18-VM -j ACCEPT -A v-9-VM -m physdev --physdev-in vnet2 --physdev-is-bridged -j RETURN -A v-9-VM -m physdev --physdev-in vnet1 --physdev-is-bridged -j RETURN -A v-9-VM -j ACCEPT

ip a gives: 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master cloudbr0 state UP group default qlen 1000 link/ether 1c:6f:65:91:9e:86 brd ff:ff:ff:ff:ff:ff altname enp4s0 4: cloudbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 1c:6f:65:91:9e:86 brd ff:ff:ff:ff:ff:ff inet 79.143.179.221/24 metric 100 brd 79.143.179.255 scope global dynamic cloudbr0 valid_lft 3105sec preferred_lft 3105sec inet6 fe80::1e6f:65ff:fe91:9e86/64 scope link valid_lft forever preferred_lft forever 5: cloudbr1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000 link/ether 1c:6f:65:91:9e:86 brd ff:ff:ff:ff:ff:ff inet6 fe80::1e6f:65ff:fe91:9e86/64 scope link valid_lft forever preferred_lft forever 7: cloud0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether a6:d6:15:d8:3a:12 brd ff:ff:ff:ff:ff:ff inet 169.254.0.1/16 scope global cloud0 valid_lft forever preferred_lft forever 40: eth0.200@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master breth0-200 state UP group default qlen 1000 link/ether 1c:6f:65:91:9e:86 brd ff:ff:ff:ff:ff:ff 41: breth0-200: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 82:8d:82:b1:a4:f2 brd ff:ff:ff:ff:ff:ff 42: vnet18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cloud0 state UNKNOWN group default qlen 1000 link/ether fe:00:a9:fe:b9:23 brd ff:ff:ff:ff:ff:ff 43: vnet19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cloudbr0 state UNKNOWN group default qlen 1000 link/ether fe:00:e3:00:01:7b brd ff:ff:ff:ff:ff:ff 44: vnet20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master breth0-200 state UNKNOWN group default qlen 1000 link/ether fe:00:14:00:00:69 brd ff:ff:ff:ff:ff:ff 45: vnet21: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cloud0 state UNKNOWN group default qlen 1000 link/ether fe:00:a9:fe:a1:f9 brd ff:ff:ff:ff:ff:ff 46: vnet22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cloudbr0 state UNKNOWN group default qlen 1000 link/ether fe:00:7d:00:01:1c brd ff:ff:ff:ff:ff:ff 47: vnet23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master breth0-200 state UNKNOWN group default qlen 1000 link/ether fe:00:ed:00:00:68 brd ff:ff:ff:ff:ff:ff

/etc/ufw/before.rules: #

rules.before

Rules that should be run before the ufw command line added rules. Custom

rules should be added to one of these chains:

ufw-before-input

ufw-before-output

ufw-before-forward

Don't delete these required lines, otherwise there will be errors

*filter :ufw-before-input - [0:0] :ufw-before-output - [0:0] :ufw-before-forward - [0:0] :ufw-not-local - [0:0]

End required lines

allow all on loopback

-A ufw-before-input -i lo -j ACCEPT -A ufw-before-output -o lo -j ACCEPT

quickly process packets for which we already have a connection

-A ufw-before-input -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A ufw-before-output -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A ufw-before-forward -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

drop INVALID packets (logs these in loglevel medium and higher)

-A ufw-before-input -m conntrack --ctstate INVALID -j ufw-logging-deny -A ufw-before-input -m conntrack --ctstate INVALID -j DROP

ok icmp codes for INPUT

-A ufw-before-input -p icmp --icmp-type destination-unreachable -j ACCEPT -A ufw-before-input -p icmp --icmp-type time-exceeded -j ACCEPT -A ufw-before-input -p icmp --icmp-type parameter-problem -j ACCEPT -A ufw-before-input -p icmp --icmp-type echo-request -j ACCEPT

ok icmp code for FORWARD

-A ufw-before-forward -p icmp --icmp-type destination-unreachable -j ACCEPT -A ufw-before-forward -p icmp --icmp-type time-exceeded -j ACCEPT -A ufw-before-forward -p icmp --icmp-type parameter-problem -j ACCEPT -A ufw-before-forward -p icmp --icmp-type echo-request -j ACCEPT

allow dhcp client to work

-A ufw-before-input -p udp --sport 67 --dport 68 -j ACCEPT

ufw-not-local

# -A ufw-before-input -j ufw-not-local

if LOCAL, RETURN

-A ufw-not-local -m addrtype --dst-type LOCAL -j RETURN

if MULTICAST, RETURN

-A ufw-not-local -m addrtype --dst-type MULTICAST -j RETURN

if BROADCAST, RETURN

-A ufw-not-local -m addrtype --dst-type BROADCAST -j RETURN

all other non-local packets are dropped

-A ufw-not-local -m limit --limit 3/min --limit-burst 10 -j ufw-logging-deny -A ufw-not-local -j DROP

allow MULTICAST mDNS for service discovery (be sure the MULTICAST line above

is uncommented)

-A ufw-before-input -p udp -d 224.0.0.251 --dport 5353 -j ACCEPT

allow MULTICAST UPnP for service discovery (be sure the MULTICAST line above

is uncommented)

-A ufw-before-input -p udp -d 239.255.255.250 --dport 1900 -j ACCEPT

-A FORWARD -d 79.143.179.221 -j ACCEPT -A FORWARD -s 79.143.179.221 -j ACCEPT

don't delete the 'COMMIT' line or these rules won't be processed

COMMIT

virsh net-list --all gives:

` Name State Autostart Persistent

default inactive yes yes `

before_rules.txt agent_properties.txt mgmt-server_properties.txt

Please let me know, if i can provide anything else!

@MilanHofmann have you restarted the mgmt server and ssvm after updating the global setting?

there should be a default route in system vms.

@weizhouapache Unfortunately no luck with that. Still no default route and when added, VMs are not reachable from outside the network.

@MilanHofmann any error in /var/log/cloud.log in the ssvm ?

@weizhouapache Yes there are some interesting lines:

line 66: 2023-02-11 19:25:22,793 WARN [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:null) addRouteToInternalIp: unable to determine same subnet: _eth1ip=null, dest ip=MGMT_Server_IP, _eth1mask=null 2023-02-11 19:25:22,815 WARN [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:null) Error in configuring route to internal ip err=Error: inet address is expected rather than "null".

vm.log

@MilanHofmann What's the value of global setting "management.network.cidr" ?

@weizhouapache it's -> 172.31.16.0/20

@MilanHofmann

the management cidr should be changed to the cidr of management server.
please upload /var/cache/cloud/cmdline in system vms
please upload some tables in database, (1) networks; (2) vlan; (3) nics

@weizhouapache I am sorry, I am not able to access the host anymore. Today I got an email from the provider of my dedicated server, that the IPs that my cloudstack host assigned (unauthorized) to the system vms caused massive network disturbance and problems for other clients. Therefore they had to suspend the access.

I guess there is no other option than switching back to basic network config and buy ips from them, right? Anyway, thank you so much for your awesome support!!!

@MilanHofmann no problem.

the "real" public ip is not mandatory for cloudstack

@weizhouapache Cool. How can I configure cloudstack to not assign the public ip if it is not needed?

@weizhouapache So I have been switching completely to AWS now and setup the host again. I am still running into connectivity issues. The system VM now has a default route, but cannot ping the host or external servers and is of course not reachable. I have already raised an issue on AWS support, they believe the configuration should be right - but they promised to come back to me with more detailed feedback. Not knowing if this is an issue related to AWS I'll just share the info required 3 days ago. Let me know, if there's more I can provide!

sudo cat /var/cache/cloud/cmdline template=domP type=secstorage host=$MGMT_SERVER_IP port=8250 name=s-19-VM zone=6 pod=5 guid=s-19-VM workers=5 authorized_key=$AUTHORIZED_KEY resource=com.cloud.storage.resource.PremiumSecondaryStorageResource instance=SecStorage sslcopy=false role=templateProcessor mtu=1500 eth2ip=172.31.0.254 eth2mask=255.255.0.0 gateway=172.31.0.1 eth0ip=169.254.109.171 eth0mask=255.255.0.0 eth1ip=172.31.0.74 eth1mask=255.255.0.0 mgmtcidr=172.31.16.0/20 localgw=172.31.0.1 private.network.device=eth1 internaldns1=8.8.8.8 dns1=$HOST_IP nfsVersion=null keystore_password=$PW

SELECT * FROM networks\G; 1. row id: 200 name: NULL uuid: e4aae368-9a88-4187-868f-d8241f305100 display_text: NULL traffic_type: Management broadcast_domain_type: Native broadcast_uri: NULL gateway: NULL cidr: NULL mode: Static network_offering_id: 2 physical_network_id: NULL data_center_id: 1 guru_name: PodBasedNetworkGuru state: Setup related: 200 domain_id: 1 account_id: 1 dns1: NULL dns2: NULL guru_data: NULL set_fields: 0 acl_type: NULL network_domain: NULL reservation_id: NULL guest_type: NULL restart_required: 0 created: 2023-02-11 14:52:40 removed: NULL specify_ip_ranges: 0 vpc_id: NULL ip6_gateway: NULL ip6_cidr: NULL network_cidr: NULL display_network: 1 network_acl_id: NULL streched_l2: 0 redundant: 0 external_id: NULL 2. row id: 201 name: NULL uuid: c2639b21-688c-4b21-a1d7-461e34c757b1 display_text: NULL traffic_type: Control broadcast_domain_type: LinkLocal broadcast_uri: NULL gateway: 169.254.0.1 cidr: 169.254.0.0/16 mode: Static network_offering_id: 3 physical_network_id: NULL data_center_id: 1 guru_name: ControlNetworkGuru state: Setup related: 201 domain_id: 1 account_id: 1 dns1: NULL dns2: NULL guru_data: NULL set_fields: 0 acl_type: NULL network_domain: NULL reservation_id: NULL guest_type: NULL restart_required: 0 created: 2023-02-11 14:52:43 removed: NULL specify_ip_ranges: 0 vpc_id: NULL ip6_gateway: NULL ip6_cidr: NULL network_cidr: NULL display_network: 1 network_acl_id: NULL streched_l2: 0 redundant: 0 external_id: NULL 3. row id: 202 name: NULL uuid: 3116f1c9-d812-4b6a-887c-0b28c9616393 display_text: NULL traffic_type: Storage broadcast_domain_type: Native broadcast_uri: NULL gateway: NULL cidr: NULL mode: Static network_offering_id: 4 physical_network_id: NULL data_center_id: 1 guru_name: StorageNetworkGuru state: Setup related: 202 domain_id: 1 account_id: 1 dns1: NULL dns2: NULL guru_data: NULL set_fields: 0 acl_type: NULL network_domain: NULL reservation_id: NULL guest_type: NULL restart_required: 0 created: 2023-02-11 14:52:43 removed: NULL specify_ip_ranges: 1 vpc_id: NULL ip6_gateway: NULL ip6_cidr: NULL network_cidr: NULL display_network: 1 network_acl_id: NULL streched_l2: 0 redundant: 0 external_id: NULL 4. row id: 203 name: defaultGuestNetwork uuid: 29d73c50-06a6-4131-a68f-436db56a9c34 display_text: defaultGuestNetwork traffic_type: Guest broadcast_domain_type: Vlan broadcast_uri: vlan://200 gateway: 79.143.179.1 cidr: 79.143.179.0/24 mode: Dhcp network_offering_id: 7 physical_network_id: 200 data_center_id: 1 guru_name: DirectNetworkGuru state: Setup related: 203 domain_id: 1 account_id: 1 dns1: NULL dns2: NULL guru_data: NULL set_fields: 0 acl_type: Domain network_domain: cs1cloud.internal reservation_id: NULL guest_type: Shared restart_required: 0 created: 2023-02-11 14:52:56 removed: NULL specify_ip_ranges: 1 vpc_id: NULL ip6_gateway: NULL ip6_cidr: NULL network_cidr: NULL display_network: 1 network_acl_id: NULL streched_l2: 0 redundant: 0 external_id: NULL 5. row id: 204 name: NULL uuid: 7a13ff25-ef4b-4c1d-8652-1cdd4fc06c76 display_text: NULL traffic_type: Management broadcast_domain_type: Native broadcast_uri: NULL gateway: NULL cidr: NULL mode: Static network_offering_id: 2 physical_network_id: NULL data_center_id: 2 guru_name: PodBasedNetworkGuru state: Setup related: 204 domain_id: 1 account_id: 1 dns1: NULL dns2: NULL guru_data: NULL set_fields: 0 acl_type: NULL network_domain: NULL reservation_id: NULL guest_type: NULL restart_required: 0 created: 2023-02-11 16:23:21 removed: NULL specify_ip_ranges: 0 vpc_id: NULL ip6_gateway: NULL ip6_cidr: NULL network_cidr: NULL display_network: 1 network_acl_id: NULL streched_l2: 0 redundant: 0 external_id: NULL 6. row id: 205 name: NULL uuid: 447ad44f-e5f1-4a1a-90f9-7a8f58fa7f6e display_text: NULL traffic_type: Control broadcast_domain_type: LinkLocal broadcast_uri: NULL gateway: 169.254.0.1 cidr: 169.254.0.0/16 mode: Static network_offering_id: 3 physical_network_id: NULL data_center_id: 2 guru_name: ControlNetworkGuru state: Setup related: 205 domain_id: 1 account_id: 1 dns1: NULL dns2: NULL guru_data: NULL set_fields: 0 acl_type: NULL network_domain: NULL reservation_id: NULL guest_type: NULL restart_required: 0 created: 2023-02-11 16:23:21 removed: NULL specify_ip_ranges: 0 vpc_id: NULL ip6_gateway: NULL ip6_cidr: NULL network_cidr: NULL display_network: 1 network_acl_id: NULL streched_l2: 0 redundant: 0 external_id: NULL 7. row id: 206 name: NULL uuid: b14d6c9b-3cbf-432d-8a95-2125c682f2c0 display_text: NULL traffic_type: Storage broadcast_domain_type: Native broadcast_uri: NULL gateway: NULL cidr: NULL mode: Static network_offering_id: 4 physical_network_id: NULL data_center_id: 2 guru_name: StorageNetworkGuru state: Setup related: 206 domain_id: 1 account_id: 1 dns1: NULL dns2: NULL guru_data: NULL set_fields: 0 acl_type: NULL network_domain: NULL reservation_id: NULL guest_type: NULL restart_required: 0 created: 2023-02-11 16:23:21 removed: NULL specify_ip_ranges: 1 vpc_id: NULL ip6_gateway: NULL ip6_cidr: NULL network_cidr: NULL display_network: 1 network_acl_id: NULL streched_l2: 0 redundant: 0 external_id: NULL 8. row id: 207 name: defaultGuestNetwork uuid: 654b4259-311a-43e7-b3e8-bf94a83f173b display_text: defaultGuestNetwork traffic_type: Guest broadcast_domain_type: Vlan broadcast_uri: vlan://200 gateway: 79.143.179.1 cidr: 79.143.179.0/24 mode: Dhcp network_offering_id: 7 physical_network_id: 201 data_center_id: 2 guru_name: DirectNetworkGuru state: Setup related: 207 domain_id: 1 account_id: 1 dns1: NULL dns2: NULL guru_data: NULL set_fields: 0 acl_type: Domain network_domain: cs1cloud.internal reservation_id: NULL guest_type: Shared restart_required: 0 created: 2023-02-11 16:23:32 removed: NULL specify_ip_ranges: 1 vpc_id: NULL ip6_gateway: NULL ip6_cidr: NULL network_cidr: NULL display_network: 1 network_acl_id: NULL streched_l2: 0 redundant: 0 external_id: NULL 9. row id: 208 name: NULL uuid: 778e8098-b394-4239-8124-bc9810862ebc display_text: NULL traffic_type: Management broadcast_domain_type: Native broadcast_uri: NULL gateway: NULL cidr: NULL mode: Static network_offering_id: 2 physical_network_id: NULL data_center_id: 3 guru_name: PodBasedNetworkGuru state: Setup related: 208 domain_id: 1 account_id: 1 dns1: NULL dns2: NULL guru_data: NULL set_fields: 0 acl_type: NULL network_domain: NULL reservation_id: NULL guest_type: NULL restart_required: 0 created: 2023-02-15 05:07:00 removed: NULL specify_ip_ranges: 0 vpc_id: NULL ip6_gateway: NULL ip6_cidr: NULL network_cidr: NULL display_network: 1 network_acl_id: NULL streched_l2: 0 redundant: 0 external_id: NULL 10. row id: 209 name: NULL uuid: fa079af7-272f-47a7-844a-92c36190aea2 display_text: NULL traffic_type: Control broadcast_domain_type: LinkLocal broadcast_uri: NULL gateway: 169.254.0.1 cidr: 169.254.0.0/16 mode: Static network_offering_id: 3 physical_network_id: NULL data_center_id: 3 guru_name: ControlNetworkGuru state: Setup related: 209 domain_id: 1 account_id: 1 dns1: NULL dns2: NULL guru_data: NULL set_fields: 0 acl_type: NULL network_domain: NULL reservation_id: NULL guest_type: NULL restart_required: 0 created: 2023-02-15 05:07:00 removed: NULL specify_ip_ranges: 0 vpc_id: NULL ip6_gateway: NULL ip6_cidr: NULL network_cidr: NULL display_network: 1 network_acl_id: NULL streched_l2: 0 redundant: 0 external_id: NULL 11. row id: 210 name: NULL uuid: 2aff16fb-9a73-4d6d-ab26-623857189685 display_text: NULL traffic_type: Storage broadcast_domain_type: Native broadcast_uri: NULL gateway: NULL cidr: NULL mode: Static network_offering_id: 4 physical_network_id: NULL data_center_id: 3 guru_name: StorageNetworkGuru state: Setup related: 210 domain_id: 1 account_id: 1 dns1: NULL dns2: NULL guru_data: NULL set_fields: 0 acl_type: NULL network_domain: NULL reservation_id: NULL guest_type: NULL restart_required: 0 created: 2023-02-15 05:07:00 removed: NULL specify_ip_ranges: 1 vpc_id: NULL ip6_gateway: NULL ip6_cidr: NULL network_cidr: NULL display_network: 1 network_acl_id: NULL streched_l2: 0 redundant: 0 external_id: NULL 12. row id: 213 name: defaultGuestNetwork uuid: c6460052-6989-4acc-a3ae-f799535c1817 display_text: defaultGuestNetwork traffic_type: Guest broadcast_domain_type: Vlan broadcast_uri: vlan://300 gateway: 172.31.59.115 cidr: 172.31.59.0/24 mode: Dhcp network_offering_id: 7 physical_network_id: 202 data_center_id: 3 guru_name: DirectNetworkGuru state: Setup related: 213 domain_id: 1 account_id: 1 dns1: NULL dns2: NULL guru_data: NULL set_fields: 0 acl_type: Domain network_domain: cs1cloud.internal reservation_id: NULL guest_type: Shared restart_required: 0 created: 2023-02-15 05:09:25 removed: NULL specify_ip_ranges: 1 vpc_id: NULL ip6_gateway: NULL ip6_cidr: NULL network_cidr: NULL display_network: 1 network_acl_id: NULL streched_l2: 0 redundant: 0 external_id: NULL 13. row id: 214 name: NULL uuid: 0250032b-ff2b-4678-a791-6312e91b7c3b display_text: NULL traffic_type: Management broadcast_domain_type: Native broadcast_uri: NULL gateway: NULL cidr: NULL mode: Static network_offering_id: 2 physical_network_id: NULL data_center_id: 4 guru_name: PodBasedNetworkGuru state: Setup related: 214 domain_id: 1 account_id: 1 dns1: NULL dns2: NULL guru_data: NULL set_fields: 0 acl_type: NULL network_domain: NULL reservation_id: NULL guest_type: NULL restart_required: 0 created: 2023-02-15 12:30:45 removed: NULL specify_ip_ranges: 0 vpc_id: NULL ip6_gateway: NULL ip6_cidr: NULL network_cidr: NULL display_network: 1 network_acl_id: NULL streched_l2: 0 redundant: 0 external_id: NULL 14. row id: 215 name: NULL uuid: c699b6de-86cf-40d2-8c3a-53969636673f display_text: NULL traffic_type: Control broadcast_domain_type: LinkLocal broadcast_uri: NULL gateway: 169.254.0.1 cidr: 169.254.0.0/16 mode: Static network_offering_id: 3 physical_network_id: NULL data_center_id: 4 guru_name: ControlNetworkGuru state: Setup related: 215 domain_id: 1 account_id: 1 dns1: NULL dns2: NULL guru_data: NULL set_fields: 0 acl_type: NULL network_domain: NULL reservation_id: NULL guest_type: NULL restart_required: 0 created: 2023-02-15 12:30:45 removed: NULL specify_ip_ranges: 0 vpc_id: NULL ip6_gateway: NULL ip6_cidr: NULL network_cidr: NULL display_network: 1 network_acl_id: NULL streched_l2: 0 redundant: 0 external_id: NULL 15. row id: 216 name: NULL uuid: e2ed499a-f06a-4bcc-9c09-33e387c8f326 display_text: NULL traffic_type: Storage broadcast_domain_type: Native broadcast_uri: NULL gateway: NULL cidr: NULL mode: Static network_offering_id: 4 physical_network_id: NULL data_center_id: 4 guru_name: StorageNetworkGuru state: Setup related: 216 domain_id: 1 account_id: 1 dns1: NULL dns2: NULL guru_data: NULL set_fields: 0 acl_type: NULL network_domain: NULL reservation_id: NULL guest_type: NULL restart_required: 0 created: 2023-02-15 12:30:45 removed: NULL specify_ip_ranges: 1 vpc_id: NULL ip6_gateway: NULL ip6_cidr: NULL network_cidr: NULL display_network: 1 network_acl_id: NULL streched_l2: 0 redundant: 0 external_id: NULL 16. row id: 217 name: NULL uuid: f6b3f3e9-bb49-4235-9d96-aaabcba78bed display_text: NULL traffic_type: Management broadcast_domain_type: Native broadcast_uri: NULL gateway: NULL cidr: NULL mode: Static network_offering_id: 2 physical_network_id: NULL data_center_id: 5 guru_name: PodBasedNetworkGuru state: Setup related: 217 domain_id: 1 account_id: 1 dns1: NULL dns2: NULL guru_data: NULL set_fields: 0 acl_type: NULL network_domain: NULL reservation_id: NULL guest_type: NULL restart_required: 0 created: 2023-02-15 13:19:47 removed: NULL specify_ip_ranges: 0 vpc_id: NULL ip6_gateway: NULL ip6_cidr: NULL network_cidr: NULL display_network: 1 network_acl_id: NULL streched_l2: 0 redundant: 0 external_id: NULL 17. row id: 218 name: NULL uuid: da9eb7b5-44da-4feb-95c5-36d7a5fe264c display_text: NULL traffic_type: Control broadcast_domain_type: LinkLocal broadcast_uri: NULL gateway: 169.254.0.1 cidr: 169.254.0.0/16 mode: Static network_offering_id: 3 physical_network_id: NULL data_center_id: 5 guru_name: ControlNetworkGuru state: Setup related: 218 domain_id: 1 account_id: 1 dns1: NULL dns2: NULL guru_data: NULL set_fields: 0 acl_type: NULL network_domain: NULL reservation_id: NULL guest_type: NULL restart_required: 0 created: 2023-02-15 13:19:47 removed: NULL specify_ip_ranges: 0 vpc_id: NULL ip6_gateway: NULL ip6_cidr: NULL network_cidr: NULL display_network: 1 network_acl_id: NULL streched_l2: 0 redundant: 0 external_id: NULL 18. row id: 219 name: NULL uuid: 33018a3d-9934-4a38-998e-04aed0b7a285 display_text: NULL traffic_type: Storage broadcast_domain_type: Native broadcast_uri: NULL gateway: NULL cidr: NULL mode: Static network_offering_id: 4 physical_network_id: NULL data_center_id: 5 guru_name: StorageNetworkGuru state: Setup related: 219 domain_id: 1 account_id: 1 dns1: NULL dns2: NULL guru_data: NULL set_fields: 0 acl_type: NULL network_domain: NULL reservation_id: NULL guest_type: NULL restart_required: 0 created: 2023-02-15 13:19:47 removed: NULL specify_ip_ranges: 1 vpc_id: NULL ip6_gateway: NULL ip6_cidr: NULL network_cidr: NULL display_network: 1 network_acl_id: NULL streched_l2: 0 redundant: 0 external_id: NULL 19. row id: 220 name: NULL uuid: d602cbae-6e19-4ed0-821d-da2ac1ac4e16 display_text: NULL traffic_type: Management broadcast_domain_type: Native broadcast_uri: NULL gateway: NULL cidr: NULL mode: Static network_offering_id: 2 physical_network_id: NULL data_center_id: 6 guru_name: PodBasedNetworkGuru state: Setup related: 220 domain_id: 1 account_id: 1 dns1: NULL dns2: NULL guru_data: NULL set_fields: 0 acl_type: NULL network_domain: NULL reservation_id: NULL guest_type: NULL restart_required: 0 created: 2023-02-15 13:31:04 removed: NULL specify_ip_ranges: 0 vpc_id: NULL ip6_gateway: NULL ip6_cidr: NULL network_cidr: NULL display_network: 1 network_acl_id: NULL streched_l2: 0 redundant: 0 external_id: NULL 20. row id: 221 name: NULL uuid: e1793d7f-4674-473b-9f3d-482ee0811e58 display_text: NULL traffic_type: Control broadcast_domain_type: LinkLocal broadcast_uri: NULL gateway: 169.254.0.1 cidr: 169.254.0.0/16 mode: Static network_offering_id: 3 physical_network_id: NULL data_center_id: 6 guru_name: ControlNetworkGuru state: Setup related: 221 domain_id: 1 account_id: 1 dns1: NULL dns2: NULL guru_data: NULL set_fields: 0 acl_type: NULL network_domain: NULL reservation_id: NULL guest_type: NULL restart_required: 0 created: 2023-02-15 13:31:05 removed: NULL specify_ip_ranges: 0 vpc_id: NULL ip6_gateway: NULL ip6_cidr: NULL network_cidr: NULL display_network: 1 network_acl_id: NULL streched_l2: 0 redundant: 0 external_id: NULL 21. row id: 222 name: NULL uuid: d4a2e8b9-02cc-4f8a-bdea-aafa63c3b777 display_text: NULL traffic_type: Storage broadcast_domain_type: Native broadcast_uri: NULL gateway: NULL cidr: NULL mode: Static network_offering_id: 4 physical_network_id: NULL data_center_id: 6 guru_name: StorageNetworkGuru state: Setup related: 222 domain_id: 1 account_id: 1 dns1: NULL dns2: NULL guru_data: NULL set_fields: 0 acl_type: NULL network_domain: NULL reservation_id: NULL guest_type: NULL restart_required: 0 created: 2023-02-15 13:31:05 removed: NULL specify_ip_ranges: 1 vpc_id: NULL ip6_gateway: NULL ip6_cidr: NULL network_cidr: NULL display_network: 1 network_acl_id: NULL streched_l2: 0 redundant: 0 external_id: NULL 22. row id: 223 name: defaultGuestNetwork uuid: 9d62e2b6-53a8-4925-97f0-c7efde9c3509 display_text: defaultGuestNetwork traffic_type: Guest broadcast_domain_type: Vlan broadcast_uri: vlan://300 gateway: 172.31.0.1 cidr: 172.31.0.0/16 mode: Dhcp network_offering_id: 7 physical_network_id: 205 data_center_id: 6 guru_name: DirectNetworkGuru state: Setup related: 223 domain_id: 1 account_id: 1 dns1: NULL dns2: NULL guru_data: NULL set_fields: 0 acl_type: Domain network_domain: cs1cloud.internal reservation_id: NULL guest_type: Shared restart_required: 0 created: 2023-02-15 13:31:14 removed: NULL specify_ip_ranges: 1 vpc_id: NULL ip6_gateway: NULL ip6_cidr: NULL network_cidr: NULL display_network: 1 network_acl_id: NULL streched_l2: 0 redundant: 0 external_id: NULL 22 rows in set (0.00 sec)

SELECT * FROM vlan\G; 1. row id: 4 uuid: 20cc10d5-2114-4943-b5b2-564b72f27e00 vlan_id: 300 vlan_gateway: 172.31.0.1 vlan_netmask: 255.255.0.0 ip4_range: 172.31.0.152-172.31.0.254 vlan_type: DirectAttached data_center_id: 6 network_id: 223 physical_network_id: 205 ip6_gateway: NULL ip6_cidr: NULL ip6_range: NULL removed: NULL created: 2023-02-15 13:31:14 1 row in set (0.00 sec)

SELECT * FROM nics\G; 1. row id: 31 uuid: 0de87ac4-cd8b-4544-89e5-8b642180ac0a instance_id: 11 mac_address: 1e:00:ef:00:00:73 ip4_address: 172.31.59.114 netmask: 255.255.255.0 gateway: 172.31.59.115 ip_type: Ip4 broadcast_uri: vlan://300 network_id: 213 mode: Dhcp state: Deallocating strategy: Create reserver_name: DirectNetworkGuru reservation_id: NULL device_id: 2 update_time: NULL isolation_uri: vlan://300 ip6_address: NULL default_nic: 1 vm_type: SecondaryStorageVm created: 2023-02-15 07:23:27 removed: 2023-02-15 11:52:32 ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 2. row id: 32 uuid: b7cbc1c1-fd3a-4761-b283-2c33f63d34d3 instance_id: 11 mac_address: NULL ip4_address: NULL netmask: NULL gateway: NULL ip_type: NULL broadcast_uri: NULL network_id: 209 mode: Static state: Deallocating strategy: Start reserver_name: ControlNetworkGuru reservation_id: 87a4bf28-cb7e-4881-9c2f-7a72910954db device_id: 0 update_time: NULL isolation_uri: NULL ip6_address: NULL default_nic: 0 vm_type: SecondaryStorageVm created: 2023-02-15 07:23:27 removed: 2023-02-15 11:52:32 ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 3. row id: 33 uuid: 3bb8ca99-43a3-48ed-bae2-f6c63efab40c instance_id: 11 mac_address: NULL ip4_address: NULL netmask: NULL gateway: NULL ip_type: NULL broadcast_uri: NULL network_id: 208 mode: Static state: Deallocating strategy: Start reserver_name: PodBasedNetworkGuru reservation_id: 87a4bf28-cb7e-4881-9c2f-7a72910954db device_id: 1 update_time: NULL isolation_uri: NULL ip6_address: NULL default_nic: 0 vm_type: SecondaryStorageVm created: 2023-02-15 07:23:27 removed: 2023-02-15 11:52:32 ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 4. row id: 34 uuid: 1439029d-5d06-45dd-b7ab-c63b61fea355 instance_id: 12 mac_address: 1e:00:d0:00:00:72 ip4_address: 172.31.59.113 netmask: 255.255.255.0 gateway: 172.31.59.115 ip_type: Ip4 broadcast_uri: vlan://300 network_id: 213 mode: Dhcp state: Deallocating strategy: Create reserver_name: DirectNetworkGuru reservation_id: NULL device_id: 2 update_time: NULL isolation_uri: vlan://300 ip6_address: NULL default_nic: 1 vm_type: ConsoleProxy created: 2023-02-15 07:23:29 removed: 2023-02-15 11:52:27 ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 5. row id: 35 uuid: a78a8f7a-6b75-4509-8b7c-02781b262413 instance_id: 12 mac_address: NULL ip4_address: NULL netmask: NULL gateway: NULL ip_type: NULL broadcast_uri: NULL network_id: 209 mode: Static state: Deallocating strategy: Start reserver_name: ControlNetworkGuru reservation_id: e06548fc-8d59-4c44-9f14-0aaaab66ba3a device_id: 0 update_time: NULL isolation_uri: NULL ip6_address: NULL default_nic: 0 vm_type: ConsoleProxy created: 2023-02-15 07:23:29 removed: 2023-02-15 11:52:27 ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 6. row id: 36 uuid: 37ff420e-9808-4ef3-ab25-73485ace9c7b instance_id: 12 mac_address: NULL ip4_address: NULL netmask: NULL gateway: NULL ip_type: NULL broadcast_uri: NULL network_id: 208 mode: Static state: Deallocating strategy: Start reserver_name: PodBasedNetworkGuru reservation_id: e06548fc-8d59-4c44-9f14-0aaaab66ba3a device_id: 1 update_time: NULL isolation_uri: NULL ip6_address: NULL default_nic: 0 vm_type: ConsoleProxy created: 2023-02-15 07:23:29 removed: 2023-02-15 11:52:28 ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 7. row id: 37 uuid: 9e1d0f62-a2b1-481b-9e2a-dc2de6a73d6e instance_id: 13 mac_address: 1e:00:4f:00:00:72 ip4_address: 172.31.59.113 netmask: 255.255.255.0 gateway: 172.31.59.115 ip_type: Ip4 broadcast_uri: vlan://300 network_id: 213 mode: Dhcp state: Deallocating strategy: Create reserver_name: DirectNetworkGuru reservation_id: NULL device_id: 2 update_time: NULL isolation_uri: vlan://300 ip6_address: NULL default_nic: 1 vm_type: ConsoleProxy created: 2023-02-15 11:52:28 removed: 2023-02-15 12:03:44 ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 8. row id: 38 uuid: 55f03a98-630f-40f4-ac26-6ccaf9a7946d instance_id: 13 mac_address: NULL ip4_address: NULL netmask: NULL gateway: NULL ip_type: NULL broadcast_uri: NULL network_id: 209 mode: Static state: Deallocating strategy: Start reserver_name: ControlNetworkGuru reservation_id: 17a103ce-e94c-4145-9307-fc5d418bdcad device_id: 0 update_time: NULL isolation_uri: NULL ip6_address: NULL default_nic: 0 vm_type: ConsoleProxy created: 2023-02-15 11:52:28 removed: 2023-02-15 12:03:44 ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 9. row id: 39 uuid: e1db8aef-446a-43f2-856d-0ca9b0e0693b instance_id: 13 mac_address: NULL ip4_address: NULL netmask: NULL gateway: NULL ip_type: NULL broadcast_uri: NULL network_id: 208 mode: Static state: Deallocating strategy: Start reserver_name: PodBasedNetworkGuru reservation_id: 17a103ce-e94c-4145-9307-fc5d418bdcad device_id: 1 update_time: NULL isolation_uri: NULL ip6_address: NULL default_nic: 0 vm_type: ConsoleProxy created: 2023-02-15 11:52:28 removed: 2023-02-15 12:03:45 ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 10. row id: 40 uuid: 949dfa2e-1443-401c-ab0c-601fd723b139 instance_id: 14 mac_address: 1e:00:12:00:00:73 ip4_address: 172.31.59.114 netmask: 255.255.255.0 gateway: 172.31.59.115 ip_type: Ip4 broadcast_uri: vlan://300 network_id: 213 mode: Dhcp state: Deallocating strategy: Create reserver_name: DirectNetworkGuru reservation_id: NULL device_id: 2 update_time: NULL isolation_uri: vlan://300 ip6_address: NULL default_nic: 1 vm_type: SecondaryStorageVm created: 2023-02-15 11:52:57 removed: 2023-02-15 12:03:44 ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 11. row id: 41 uuid: 4209222c-65af-4433-afb7-645eaf296aea instance_id: 14 mac_address: NULL ip4_address: NULL netmask: NULL gateway: NULL ip_type: NULL broadcast_uri: NULL network_id: 209 mode: Static state: Deallocating strategy: Start reserver_name: ControlNetworkGuru reservation_id: 71a72991-4a0b-4c7a-9446-da7c348e96cf device_id: 0 update_time: NULL isolation_uri: NULL ip6_address: NULL default_nic: 0 vm_type: SecondaryStorageVm created: 2023-02-15 11:52:57 removed: 2023-02-15 12:03:44 ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 12. row id: 42 uuid: 99f42d64-87ba-42b5-a7e7-55e609a7d07f instance_id: 14 mac_address: NULL ip4_address: NULL netmask: NULL gateway: NULL ip_type: NULL broadcast_uri: NULL network_id: 208 mode: Static state: Deallocating strategy: Start reserver_name: PodBasedNetworkGuru reservation_id: 71a72991-4a0b-4c7a-9446-da7c348e96cf device_id: 1 update_time: NULL isolation_uri: NULL ip6_address: NULL default_nic: 0 vm_type: SecondaryStorageVm created: 2023-02-15 11:52:57 removed: 2023-02-15 12:03:44 ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 13. row id: 43 uuid: 93763441-f86c-44fb-abaf-b965c9a03ec2 instance_id: 15 mac_address: 1e:00:f4:00:00:73 ip4_address: 172.31.59.114 netmask: 255.255.255.0 gateway: 172.31.59.115 ip_type: Ip4 broadcast_uri: vlan://300 network_id: 213 mode: Dhcp state: Deallocating strategy: Create reserver_name: DirectNetworkGuru reservation_id: NULL device_id: 2 update_time: NULL isolation_uri: vlan://300 ip6_address: NULL default_nic: 1 vm_type: SecondaryStorageVm created: 2023-02-15 12:07:57 removed: 2023-02-15 12:23:00 ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 14. row id: 44 uuid: 9e25e15a-ee34-4770-9b17-5811eb707422 instance_id: 15 mac_address: NULL ip4_address: NULL netmask: NULL gateway: NULL ip_type: NULL broadcast_uri: NULL network_id: 209 mode: Static state: Deallocating strategy: Start reserver_name: ControlNetworkGuru reservation_id: 79926d47-1302-4566-9b9c-d43255d3c2d2 device_id: 0 update_time: NULL isolation_uri: NULL ip6_address: NULL default_nic: 0 vm_type: SecondaryStorageVm created: 2023-02-15 12:07:57 removed: 2023-02-15 12:23:00 ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 15. row id: 45 uuid: eb08be39-e914-45f5-9afe-bf7c20f8b820 instance_id: 15 mac_address: NULL ip4_address: NULL netmask: NULL gateway: NULL ip_type: NULL broadcast_uri: NULL network_id: 208 mode: Static state: Deallocating strategy: Start reserver_name: PodBasedNetworkGuru reservation_id: 79926d47-1302-4566-9b9c-d43255d3c2d2 device_id: 1 update_time: NULL isolation_uri: NULL ip6_address: NULL default_nic: 0 vm_type: SecondaryStorageVm created: 2023-02-15 12:07:57 removed: 2023-02-15 12:23:00 ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 16. row id: 46 uuid: ad787cfc-30d8-4736-a99f-098620d0e42d instance_id: 16 mac_address: 1e:00:85:00:00:72 ip4_address: 172.31.59.113 netmask: 255.255.255.0 gateway: 172.31.59.115 ip_type: Ip4 broadcast_uri: vlan://300 network_id: 213 mode: Dhcp state: Deallocating strategy: Create reserver_name: DirectNetworkGuru reservation_id: NULL device_id: 2 update_time: NULL isolation_uri: vlan://300 ip6_address: NULL default_nic: 1 vm_type: ConsoleProxy created: 2023-02-15 12:07:59 removed: 2023-02-15 12:23:00 ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 17. row id: 47 uuid: fd95efe6-9238-49f3-a65d-c626ae63e896 instance_id: 16 mac_address: NULL ip4_address: NULL netmask: NULL gateway: NULL ip_type: NULL broadcast_uri: NULL network_id: 209 mode: Static state: Deallocating strategy: Start reserver_name: ControlNetworkGuru reservation_id: cdd74ac8-c6d0-4f7f-9ad9-1d944516a10f device_id: 0 update_time: NULL isolation_uri: NULL ip6_address: NULL default_nic: 0 vm_type: ConsoleProxy created: 2023-02-15 12:07:59 removed: 2023-02-15 12:23:00 ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 18. row id: 48 uuid: 21f3a9b4-b79a-47e5-b75f-51778a923581 instance_id: 16 mac_address: NULL ip4_address: NULL netmask: NULL gateway: NULL ip_type: NULL broadcast_uri: NULL network_id: 208 mode: Static state: Deallocating strategy: Start reserver_name: PodBasedNetworkGuru reservation_id: cdd74ac8-c6d0-4f7f-9ad9-1d944516a10f device_id: 1 update_time: NULL isolation_uri: NULL ip6_address: NULL default_nic: 0 vm_type: ConsoleProxy created: 2023-02-15 12:07:59 removed: 2023-02-15 12:23:00 ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 19. row id: 49 uuid: e11d8432-bbd0-446b-8d19-82f03fb8c1e6 instance_id: 17 mac_address: 1e:00:29:00:00:67 ip4_address: 172.31.0.254 netmask: 255.255.0.0 gateway: 172.31.0.1 ip_type: Ip4 broadcast_uri: vlan://300 network_id: 223 mode: Dhcp state: Deallocating strategy: Create reserver_name: DirectNetworkGuru reservation_id: NULL device_id: 2 update_time: NULL isolation_uri: vlan://300 ip6_address: NULL default_nic: 1 vm_type: SecondaryStorageVm created: 2023-02-15 13:33:57 removed: 2023-02-15 17:51:19 ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 20. row id: 50 uuid: 83d29750-20a0-42d6-a7da-de41e22d362f instance_id: 17 mac_address: NULL ip4_address: NULL netmask: NULL gateway: NULL ip_type: NULL broadcast_uri: NULL network_id: 221 mode: Static state: Deallocating strategy: Start reserver_name: ControlNetworkGuru reservation_id: 628f2d5d-65b1-400d-8559-9f3e1ffb0083 device_id: 0 update_time: NULL isolation_uri: NULL ip6_address: NULL default_nic: 0 vm_type: SecondaryStorageVm created: 2023-02-15 13:33:57 removed: 2023-02-15 17:51:19 ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 21. row id: 51 uuid: 723231fc-8974-480e-a704-c1e16e04a792 instance_id: 17 mac_address: NULL ip4_address: NULL netmask: NULL gateway: NULL ip_type: NULL broadcast_uri: NULL network_id: 220 mode: Static state: Deallocating strategy: Start reserver_name: PodBasedNetworkGuru reservation_id: 628f2d5d-65b1-400d-8559-9f3e1ffb0083 device_id: 1 update_time: NULL isolation_uri: NULL ip6_address: NULL default_nic: 0 vm_type: SecondaryStorageVm created: 2023-02-15 13:33:57 removed: 2023-02-15 17:51:19 ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 22. row id: 52 uuid: 1189510b-63d2-48da-8718-cdb5fac4f1d3 instance_id: 18 mac_address: 1e:00:4a:00:00:66 ip4_address: 172.31.0.253 netmask: 255.255.0.0 gateway: 172.31.0.1 ip_type: Ip4 broadcast_uri: vlan://300 network_id: 223 mode: Dhcp state: Deallocating strategy: Create reserver_name: DirectNetworkGuru reservation_id: NULL device_id: 2 update_time: NULL isolation_uri: vlan://300 ip6_address: NULL default_nic: 1 vm_type: ConsoleProxy created: 2023-02-15 13:33:59 removed: 2023-02-15 17:51:22 ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 23. row id: 53 uuid: 2540f89b-c258-4a27-9120-a95fab907854 instance_id: 18 mac_address: NULL ip4_address: NULL netmask: NULL gateway: NULL ip_type: NULL broadcast_uri: NULL network_id: 221 mode: Static state: Deallocating strategy: Start reserver_name: ControlNetworkGuru reservation_id: cad30fec-a4eb-4b61-a6db-07bfb54ccc29 device_id: 0 update_time: NULL isolation_uri: NULL ip6_address: NULL default_nic: 0 vm_type: ConsoleProxy created: 2023-02-15 13:33:59 removed: 2023-02-15 17:51:22 ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 24. row id: 54 uuid: 1cd41456-ca09-4728-a1d6-0e3f25965a45 instance_id: 18 mac_address: NULL ip4_address: NULL netmask: NULL gateway: NULL ip_type: NULL broadcast_uri: NULL network_id: 220 mode: Static state: Deallocating strategy: Start reserver_name: PodBasedNetworkGuru reservation_id: cad30fec-a4eb-4b61-a6db-07bfb54ccc29 device_id: 1 update_time: NULL isolation_uri: NULL ip6_address: NULL default_nic: 0 vm_type: ConsoleProxy created: 2023-02-15 13:33:59 removed: 2023-02-15 17:51:22 ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 25. row id: 55 uuid: bf4d4e66-cdbf-41b8-9bb9-97802182c154 instance_id: 19 mac_address: 1e:00:73:00:00:67 ip4_address: 172.31.0.254 netmask: 255.255.0.0 gateway: 172.31.0.1 ip_type: Ip4 broadcast_uri: vlan://300 network_id: 223 mode: Dhcp state: Reserved strategy: Create reserver_name: DirectNetworkGuru reservation_id: NULL device_id: 2 update_time: NULL isolation_uri: vlan://300 ip6_address: NULL default_nic: 1 vm_type: SecondaryStorageVm created: 2023-02-15 17:51:57 removed: NULL ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 26. row id: 56 uuid: 95e41358-65af-4d3a-8255-2d69f2d94fbc instance_id: 19 mac_address: 0e:00:a9:fe:6d:ab ip4_address: 169.254.109.171 netmask: 255.255.0.0 gateway: 169.254.0.1 ip_type: Ip4 broadcast_uri: NULL network_id: 221 mode: Static state: Reserved strategy: Start reserver_name: ControlNetworkGuru reservation_id: 187ba359-0a00-40b4-b4a5-a0bf012238bf device_id: 0 update_time: NULL isolation_uri: NULL ip6_address: NULL default_nic: 0 vm_type: SecondaryStorageVm created: 2023-02-15 17:51:57 removed: NULL ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 27. row id: 57 uuid: 7514e057-3e6b-4e03-9354-7468a01710c6 instance_id: 19 mac_address: 1e:00:a0:00:00:ae ip4_address: 172.31.0.74 netmask: 255.255.0.0 gateway: 172.31.0.1 ip_type: Ip4 broadcast_uri: NULL network_id: 220 mode: Static state: Reserved strategy: Start reserver_name: PodBasedNetworkGuru reservation_id: 187ba359-0a00-40b4-b4a5-a0bf012238bf device_id: 1 update_time: NULL isolation_uri: NULL ip6_address: NULL default_nic: 0 vm_type: SecondaryStorageVm created: 2023-02-15 17:51:57 removed: NULL ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 28. row id: 58 uuid: 4d05d422-a827-45a2-af1e-be9bbbe752ea instance_id: 20 mac_address: 1e:00:51:00:00:66 ip4_address: 172.31.0.253 netmask: 255.255.0.0 gateway: 172.31.0.1 ip_type: Ip4 broadcast_uri: vlan://300 network_id: 223 mode: Dhcp state: Reserved strategy: Create reserver_name: DirectNetworkGuru reservation_id: NULL device_id: 2 update_time: NULL isolation_uri: vlan://300 ip6_address: NULL default_nic: 1 vm_type: ConsoleProxy created: 2023-02-15 17:51:59 removed: NULL ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 29. row id: 59 uuid: b0d68a61-ddf8-4b91-b13a-162b0a813dc1 instance_id: 20 mac_address: 0e:00:a9:fe:23:ac ip4_address: 169.254.35.172 netmask: 255.255.0.0 gateway: 169.254.0.1 ip_type: Ip4 broadcast_uri: NULL network_id: 221 mode: Static state: Reserved strategy: Start reserver_name: ControlNetworkGuru reservation_id: 70cb1da2-6523-4b3e-b1f6-ea493e4a73f2 device_id: 0 update_time: NULL isolation_uri: NULL ip6_address: NULL default_nic: 0 vm_type: ConsoleProxy created: 2023-02-15 17:51:59 removed: NULL ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 30. row id: 60 uuid: 5f22430d-39f7-44a9-82a8-48f532fd9973 instance_id: 20 mac_address: 1e:00:75:00:00:d5 ip4_address: 172.31.0.113 netmask: 255.255.0.0 gateway: 172.31.0.1 ip_type: Ip4 broadcast_uri: NULL network_id: 220 mode: Static state: Reserved strategy: Start reserver_name: PodBasedNetworkGuru reservation_id: 70cb1da2-6523-4b3e-b1f6-ea493e4a73f2 device_id: 1 update_time: NULL isolation_uri: NULL ip6_address: NULL default_nic: 0 vm_type: ConsoleProxy created: 2023-02-15 17:51:59 removed: NULL ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 30 rows in set (0.01 sec)

Also in SSVM:

ip route show default via 172.31.0.1 dev eth2 8.8.8.8 via 172.31.0.1 dev eth1 $MGMT_SERVER_IP via 172.31.0.1 dev eth1 169.254.0.0/16 dev eth0 proto kernel scope link src 169.254.109.171 172.31.0.0/16 dev eth1 proto kernel scope link src 172.31.0.74 172.31.0.0/16 dev eth2 proto kernel scope link src 172.31.0.254 172.31.16.0/20 via 172.31.0.1 dev eth1

ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 0e:00:a9:fe:6d:ab brd ff:ff:ff:ff:ff:ff altname enp0s3 altname ens3 inet 169.254.109.171/16 brd 169.254.255.255 scope global eth0 valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 1e:00:a0:00:00:ae brd ff:ff:ff:ff:ff:ff altname enp0s4 altname ens4 inet 172.31.0.74/16 brd 172.31.255.255 scope global eth1 valid_lft forever preferred_lft forever 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 1e:00:73:00:00:67 brd ff:ff:ff:ff:ff:ff altname enp0s5 altname ens5 inet 172.31.0.254/16 brd 172.31.255.255 scope global eth2 valid_lft forever preferred_lft forever

on Host ip route show default via 172.31.48.1 dev cloudbr0 proto dhcp src 172.31.59.115 metric 100 169.254.0.0/16 dev cloud0 proto kernel scope link src 169.254.0.1 172.31.0.2 via 172.31.48.1 dev cloudbr0 proto dhcp src 172.31.59.115 metric 100 172.31.48.0/20 dev cloudbr0 proto kernel scope link src 172.31.59.115 metric 100 172.31.48.1 dev cloudbr0 proto dhcp scope link src 172.31.59.115 metric 100

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 0e:00:a9:fe:6d:ab brd ff:ff:ff:ff:ff:ff altname enp0s3 altname ens3 inet 169.254.109.171/16 brd 169.254.255.255 scope global eth0 valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 1e:00:a0:00:00:ae brd ff:ff:ff:ff:ff:ff altname enp0s4 altname ens4 inet 172.31.0.74/16 brd 172.31.255.255 scope global eth1 valid_lft forever preferred_lft forever 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 1e:00:73:00:00:67 brd ff:ff:ff:ff:ff:ff altname enp0s5 altname ens5 inet 172.31.0.254/16 brd 172.31.255.255 scope global eth2 valid_lft forever preferred_lft forever

AWS support says they neither understand cloudstack or KVM well enough to check the networking issue properly.

@MilanHofmann can you turn off the firewall on kvm hosts ?

@weizhouapache it's already turned off.

@MilanHofmann sorry I am a bit confused now. what's the current status ?

It seems the SSVM has correct routes.

@weizhouapache Sure: Currently the SSVM does not have any access to the network of the host OR other servers from outside. It HAS a default route now, but no traffic goes out. I can SSH from the host into the VM, but not from the managament server. The VM is also of course not able to connect to the secondary storage. The firewall is turned off.

Good to know, that the routes are not a problem.

Maybe this will help us, the iptables of the host: -P INPUT ACCEPT -P FORWARD ACCEPT -P OUTPUT ACCEPT -N BF-brenp125s0-300 -N BF-brenp125s0-300-IN -N BF-brenp125s0-300-OUT -N BF-cloudbr0 -N BF-cloudbr0-IN -N BF-cloudbr0-OUT -N s-23-VM -N s-25-VM -N v-24-VM -N v-26-VM -A FORWARD -i cloudbr0 -o cloudbr0 -j ACCEPT -A FORWARD -o brenp125s0-300 -m physdev --physdev-is-bridged -j BF-brenp125s0-300 -A FORWARD -i brenp125s0-300 -m physdev --physdev-is-bridged -j BF-brenp125s0-300 -A FORWARD -o brenp125s0-300 -j DROP -A FORWARD -i brenp125s0-300 -j DROP -A FORWARD -o cloudbr0 -m physdev --physdev-is-bridged -j BF-cloudbr0 -A FORWARD -i cloudbr0 -m physdev --physdev-is-bridged -j BF-cloudbr0 -A FORWARD -o cloudbr0 -j DROP -A FORWARD -i cloudbr0 -j DROP -A BF-brenp125s0-300 -m state --state RELATED,ESTABLISHED -j ACCEPT -A BF-brenp125s0-300 -m physdev --physdev-is-in --physdev-is-bridged -j BF-brenp125s0-300-IN -A BF-brenp125s0-300 -m physdev --physdev-is-out --physdev-is-bridged -j BF-brenp125s0-300-OUT -A BF-brenp125s0-300 -m physdev --physdev-out enp125s0.300 --physdev-is-bridged -j ACCEPT -A BF-brenp125s0-300-IN -m physdev --physdev-in vnet8 --physdev-is-bridged -j s-25-VM -A BF-brenp125s0-300-IN -m physdev --physdev-in vnet11 --physdev-is-bridged -j v-26-VM -A BF-brenp125s0-300-OUT -m physdev --physdev-out vnet8 --physdev-is-bridged -j s-25-VM -A BF-brenp125s0-300-OUT -m physdev --physdev-out vnet11 --physdev-is-bridged -j v-26-VM -A BF-cloudbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT -A BF-cloudbr0 -m physdev --physdev-is-in --physdev-is-bridged -j BF-cloudbr0-IN -A BF-cloudbr0 -m physdev --physdev-is-out --physdev-is-bridged -j BF-cloudbr0-OUT -A BF-cloudbr0 -m physdev --physdev-out enp125s0 --physdev-is-bridged -j ACCEPT -A BF-cloudbr0-IN -m physdev --physdev-in vnet7 --physdev-is-bridged -j s-25-VM -A BF-cloudbr0-IN -m physdev --physdev-in vnet10 --physdev-is-bridged -j v-26-VM -A BF-cloudbr0-OUT -m physdev --physdev-out vnet7 --physdev-is-bridged -j s-25-VM -A BF-cloudbr0-OUT -m physdev --physdev-out vnet10 --physdev-is-bridged -j v-26-VM -A s-23-VM -m physdev --physdev-in vnet4 --physdev-is-bridged -j RETURN -A s-23-VM -m physdev --physdev-in vnet5 --physdev-is-bridged -j RETURN -A s-23-VM -j ACCEPT -A s-25-VM -m physdev --physdev-in vnet7 --physdev-is-bridged -j RETURN -A s-25-VM -m physdev --physdev-in vnet8 --physdev-is-bridged -j RETURN -A s-25-VM -j ACCEPT -A v-24-VM -m physdev --physdev-in vnet1 --physdev-is-bridged -j RETURN -A v-24-VM -m physdev --physdev-in vnet2 --physdev-is-bridged -j RETURN -A v-24-VM -j ACCEPT -A v-26-VM -m physdev --physdev-in vnet11 --physdev-is-bridged -j RETURN -A v-26-VM -m physdev --physdev-in vnet10 --physdev-is-bridged -j RETURN -A v-26-VM -j ACCEPT

@MilanHofmann You can only SSH into the system vms via link local Ip (169.254.0.x) from host. it is normal.

where is the gateway of management network (172.31.0.1) ?

@weizhouapache Ah thank you for the info.

It's the base of the VPC IPv4 network range (+1) the subnet of the ec2 instance is in. (VPC CIDR is 172.31.0.0/16).

@weizhouapache I have opened an issue for libvirt and i got this reply: """ So in your VM you have 2 interfaces which are in the same subnet:

eth1: inet 172.31.0.74/16 eth2: inet 172.31.0.254/16 But those two interfaces are connected to different bridges in the host: eth2(enp0s5) is connected to brenp125s0-300 on the host while eth1(enp0s4) is conencted to cloudbr0 Your default route in the vm also goes via eth2, but on your host via cloudbr0. Is your IP configuration manual? If so I suspect that it's wrong. I'd suggest removing the interface connected to brenp125s0-300 """ here's the issue -> https://gitlab.com/libvirt/libvirt/-/issues/447

is that a hint maybe?

@MilanHofmann yes, it could be the root cause. You should not use same IP ranges on 2 linux bridges.

I am not clear about your network topology. In my opinion, it is 99% a misconfiguration issue, not a cloudstack/libvirt issue.

It would be good to give details information of your environment. interfaces, ips, bridges, public ip ranges, management ip ranges, kvm network labels, etc

@weizhouapache Yes I am also pretty sure there must be a mistake in my configuration.

Sure, here's the network + route info: host.txt vm.txt

Netplan config: netplan.txt

virsh net-list --all Name State Autostart Persistent default inactive no yes

brctl show bridge name bridge id STP enabled interfaces brenp125s0-300 8000.6a5f361742ec no enp125s0.300 vnet2 vnet5 cloud0 8000.beb00702d583 no vnet0 vnet3 cloudbr0 8000.0e3e89a40c8d no enp125s0 vnet1 vnet4 cloudbr1 8000.0e3e89a40c8d no enp125s0.200

Anything else I can provide?

@MilanHofmann just a suspicion, should the netmask be 255.255.255.0, not 255.255.0.0 ?

@weizhouapache

@MilanHofmann just a suspicion, should the netmask be 255.255.255.0, not 255.255.0.0 ?

Just tried it out, still no success unfortunately.

@MilanHofmann can you use the cloudbr0 subnet (172.31.48.1/20) as the pod systemvm ip ?

@MilanHofmann can you use the cloudbr0 subnet (172.31.48.1/20) as the pod systemvm ip ?

yes, that's possible.

@weizhouapache I have adjusted the netmask to /20, now everything is in the same subnet and I can reach my host from inside the vm. Still, i cannot ping any external server from inside the vm, neither via eth1 or eth2. What I do not understand is, that the host is also using cloudbr0 and can connect to any server. Is there maybe a problem hidden in my iptables, and the bridge config itself is fine?

I'll append the current status: iptables.txt vm.txt host.txt

@weizhouapache I have adjusted the netmask to /20, now everything is in the same subnet and I can reach my host from inside the vm. Still, i cannot ping any external server from inside the vm, neither via eth1 or eth2. What I do not understand is, that the host is also using cloudbr0 and can connect to any server. Is there maybe a problem hidden in my iptables, and the bridge config itself is fine?

I'll append the current status: iptables.txt vm.txt host.txt

@MilanHofmann the vm has default route (via public interface eth2)

default via 172.31.49.0 dev eth2

and host

default via 172.31.48.1 dev cloudbr0 proto dhcp src 172.31.59.115 metric 100

please check if 172.31.49.0 is valid gateway.

@weizhouapache Following the instructions from libvirt-support (https://gitlab.com/libvirt/libvirt/-/issues/447#note_1284184685), I have changed the config to the following:

default via 192.168.64.1 dev eth2 8.8.8.8 via 192.168.48.1 dev eth1 18.184.181.49 via 192.168.48.1 dev eth1 169.254.0.0/16 dev eth0 proto kernel scope link src 169.254.181.44 172.31.16.0/20 via 192.168.48.1 dev eth1 192.168.48.0/20 dev eth1 proto kernel scope link src 192.168.48.8 192.168.64.0/20 dev eth2 proto kernel scope link src 192.168.64.2 root@s-41-VM:~# ping 172.31.59.115

while my guest VM xml has the following interfaces:

<interface type='bridge'>
  <mac address='0e:00:a9:fe:b5:2c'/>
  <source bridge='cloud0'/>
  <target dev='vnet0'/>
  <model type='virtio'/>
  <link state='up'/>
  <alias name='net0'/>
  <rom bar='off' file=''/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='bridge'>
  <mac address='1e:00:73:00:00:fa'/>
  <source bridge='cloudbr0'/>
  <target dev='vnet1'/>
  <model type='virtio'/>
  <link state='up'/>
  <alias name='net1'/>
  <rom bar='off' file=''/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</interface>
<interface type='bridge'>
  <mac address='1e:00:f8:00:00:01'/>
  <source bridge='brenp125s0-300'/>
  <target dev='vnet2'/>
  <model type='virtio'/>
  <link state='up'/>
  <alias name='net2'/>
  <rom bar='off' file=''/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>

while ip a on the host gives:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp125s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master cloudbr0 state UP group default qlen 1000 link/ether 0e:3e:89:a4:0c:8d brd ff:ff:ff:ff:ff:ff 3: cloudbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default qlen 1000 link/ether 0e:3e:89:a4:0c:8d brd ff:ff:ff:ff:ff:ff inet 172.31.59.115/20 metric 100 brd 172.31.63.255 scope global dynamic cloudbr0 valid_lft 2520sec preferred_lft 2520sec inet6 fe80::c3e:89ff:fea4:c8d/64 scope link valid_lft forever preferred_lft forever 4: cloudbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 0e:3e:89:a4:0c:8d brd ff:ff:ff:ff:ff:ff inet6 fe80::c3e:89ff:fea4:c8d/64 scope link valid_lft forever preferred_lft forever 5: enp125s0.200@enp125s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cloudbr1 state UP group default qlen 1000 link/ether 0e:3e:89:a4:0c:8d brd ff:ff:ff:ff:ff:ff 6: cloud0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether be:b0:07:02:d5:83 brd ff:ff:ff:ff:ff:ff inet 169.254.0.1/16 scope global cloud0 valid_lft forever preferred_lft forever inet6 fe80::bcb0:7ff:fe02:d583/64 scope link valid_lft forever preferred_lft forever 23: enp125s0.300@enp125s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master brenp125s0-300 state UP group default qlen 1000 link/ether 0e:3e:89:a4:0c:8d brd ff:ff:ff:ff:ff:ff 24: brenp125s0-300: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 6a:5f:36:17:42:ec brd ff:ff:ff:ff:ff:ff 25: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cloud0 state UNKNOWN group default qlen 1000 link/ether fe:00:a9:fe:b5:2c brd ff:ff:ff:ff:ff:ff inet6 fe80::fc00:a9ff:fefe:b52c/64 scope link valid_lft forever preferred_lft forever 26: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue master cloudbr0 state UNKNOWN group default qlen 1000 link/ether fe:00:73:00:00:fa brd ff:ff:ff:ff:ff:ff inet6 fe80::fc00:73ff:fe00:fa/64 scope link valid_lft forever preferred_lft forever 27: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master brenp125s0-300 state UNKNOWN group default qlen 1000 link/ether fe:00:f8:00:00:01 brd ff:ff:ff:ff:ff:ff inet6 fe80::fc00:f8ff:fe00:1/64 scope link valid_lft forever preferred_lft forever 28: vnet3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cloud0 state UNKNOWN group default qlen 1000 link/ether fe:00:a9:fe:fd:fd brd ff:ff:ff:ff:ff:ff inet6 fe80::fc00:a9ff:fefe:fdfd/64 scope link valid_lft forever preferred_lft forever 29: vnet4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue master cloudbr0 state UNKNOWN group default qlen 1000 link/ether fe:00:83:00:01:cd brd ff:ff:ff:ff:ff:ff inet6 fe80::fc00:83ff:fe00:1cd/64 scope link valid_lft forever preferred_lft forever 30: vnet5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master brenp125s0-300 state UNKNOWN group default qlen 1000 link/ether fe:00:96:00:00:02 brd ff:ff:ff:ff:ff:ff inet6 fe80::fc00:96ff:fe00:2/64 scope link valid_lft forever preferred_lft forever

while the hosts ip routes are: default via 172.31.48.1 dev cloudbr0 proto dhcp src 172.31.59.115 metric 100 169.254.0.0/16 dev cloud0 proto kernel scope link src 169.254.0.1 172.31.0.2 via 172.31.48.1 dev cloudbr0 proto dhcp src 172.31.59.115 metric 100 172.31.48.0/20 dev cloudbr0 proto kernel scope link src 172.31.59.115 metric 100 172.31.48.1 dev cloudbr0 proto dhcp scope link src 172.31.59.115 metric 100

netplan on host:

network: version: 2 renderer: networkd ethernets: enp125s0: match: macaddress: 0e:3e:89:a4:0c:8d vlans: enp125s0.200: id: 200 link: enp125s0 bridges: cloudbr0: dhcp4: true macaddress: 0e:3e:89:a4:0c:8d interfaces:

enp125s0 routes:
to: 0.0.0.0/0 via: 192.168.42.1/20 on-link: true nameservers: search: [ invalid ] addresses:
- 8.8.8.8
- 8.8.4.4 cloudbr1: dhcp4: true macaddress: 0e:3e:89:a4:0c:8d interfaces:
enp125s0.200

Please let me know what you mean, by "valid gateway". Or can you tell by looking at my config if it is not valid?

It's just strange because the vm is only reachable via cloud0-bridge. What did I misconfigure with cloudbr0?

The question is, where is 172.31.49.0 ? is it working ?

system vms can only be accessible via cloud0. There are some firewall rules.

Please let me know what you mean, by "valid gateway". Or can you tell by looking at my config if it is not valid?

It's just strange because the vm is only reachable via cloud0-bridge. What did I misconfigure with cloudbr0?

@weizhouapache I cannot ping it, but i can reach the other guest vms (console proxy) via the gateways.

@MilanHofmann I am closing this issue please reopen or create new issue if you feel this is invalid

apache / cloudstack

Configuring management server on ec2 instance to send external IP to host #7185

ISSUE TYPE

COMPONENT NAME

CLOUDSTACK VERSION

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

STEPS TO REPRODUCE

ACTUAL RESULTS

Good: DNS resolves cloudstack.apache.org

Management server is 172.31.28.117. Checking connectivity. 2023/02/09 20:01:11 socat[3620] E connecting to AF=2 172.31.28.117:8250: Connection timed out Good: Can connect to management server 172.31.28.117 port 8250

Good: DNS resolves cloudstack.apache.org

Management server is $externalIP. Checking connectivity. 2023/02/10 06:30:16 socat[3407] E connect(5, AF=2 $externalIP:8250, 16): Network is unreachable Good: Can connect to management server $externalIP port 8250

Good: Java process is running