Test internet connectivity of the machines

[robertvolkmann]

• Added targets to ssh into the firewall and machine in both flavors
• Added target to curl against cloudflare
• Use frr container for internet vrf also for cumulus

[robertvolkmann]

@Gerrit91 do we still need these lines in the integration test script?

echo "Adding route to leaf01 and leaf02"
make route

echo "Check if SSH login to firewall works"
# FIXME: Again this is unstable in CI integration tests
# ssh -o StrictHostKeyChecking=no metal@100.255.254.1 -C exit

[robertvolkmann]

ssh login over link-local addresses is prohibited because these are not part of hosts.allowed anymore.

@majst01 Any reason not to include link-local addresses?

[majst01]

ssh login over link-local addresses is prohibited because these are not part of hosts.allowed anymore.

@majst01 Any reason not to include link-local addresses?

This was part of the requirements from the CIS benchmark. I think we can allow ssh access from localhost.

[robertvolkmann]

Do we need the SSH access only for the integration tests?

[majst01]

Do we need the SSH access only for the integration tests?

It is actually in the hosts.allow:

ALL: [fc00::]/7
ALL: 10.0.0.0/8
ALL: 100.64.0.0/10
ALL: 172.16.0.0/12
ALL: 192.168.0.0/16
ALL: 127.0.0.0/8

Is the ipv6 link local missing ?

[robertvolkmann]

Yes, the LLA range fe80::/10 is not contained within the ULA range fc00::/7. Adding the LLA range to hosts.allow would allow SSH connections from the leaves to the machines.

[majst01]

Yes, the LLA range fe80::/10 is not contained within the ULA range fc00::/7. Adding the LLA range to hosts.allow would allow SSH connections from the leaves to the machines.

But this is not intended for production, can we modify this only in the integration tests in mini-lab ?

[robertvolkmann]

Can I change hosts.allow by providing some ignition file with --userdata or should I use the serial console?

[majst01]

Can I change hosts.allow by providing some ignition file with --userdata or should I use the serial console?

Userdata should be the simplest way as long as you only test the metal-only use case. With gardener the userdata is generated by os-metal-extension

[robertvolkmann]

The firewall has access to the internet, but the machine does not. Packets arrive at the firewall but do not enter the vrf104009.

root@fw:~# tcpdump -i any -n icmp
13:53:49.074232 vni20 P   IP 10.0.0.2 > 1.1.1.1: ICMP echo request, id 17, seq 257, length 64
13:53:49.074232 bridge In  IP 10.0.0.2 > 1.1.1.1: ICMP echo request, id 17, seq 257, length 64
13:53:49.074232 vlan20 In  IP 10.0.0.2 > 1.1.1.1: ICMP echo request, id 17, seq 257, length 64

But the route leaking looks good.

fw# show ip route vrf vrf20
Codes: K - kernel route, C - connected, L - local, S - static,
       R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric, t - Table-Direct,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

VRF vrf20:
B>* 0.0.0.0/0 [20/0] via 10.0.0.21, vlan104009 (vrf vrf104009) onlink, weight 1, 00:04:40
L * 10.0.0.1/32 is directly connected, vlan20, 01:42:37
C>* 10.0.0.1/32 is directly connected, vlan20, 01:42:37
B>* 10.0.0.2/32 [20/0] via 10.0.0.11, vlan20 onlink, weight 1, 00:04:40
  *                    via 10.0.0.12, vlan20 onlink, weight 1, 00:04:40

[robertvolkmann]

Firewall images not containing the PR https://github.com/metal-stack/metal-images/pull/214 work on my machine. But outgoing access still doesn't work on our CI system.

@majst01 Any workaround for the PR?

[majst01]

Firewall images not containing the PR metal-stack/metal-images#214 work on my machine. But outgoing access still doesn't work on our CI system.

@majst01 Any workaround for the PR?

The firewall creation in the mini-lab must contain a initial ruleset to allow traffic, see metalctl firewall create -h

     --firewall-rules-file string   firewall rules specified in a yaml file                                                                                                                                                                                                               

                                     Example:                                                                                                                                                                                                                                              

                                     $ metalctl firewall create ..mandatory args.. --firewall-rules-file rules.yaml                                                                                                                                                                        

                                     rules.yaml                                                                                                                                                                                                                                            
                                     ---                                                                                                                                                                                                                                                   
                                     egress:                                                                                                                                                                                                                                               
                                     - comment: allow outgoing https                                                                                                                                                                                                                       
                                       ports:                                                                                                                                                                                                                                              
                                       - 443                                                                                                                                                                                                                                               
                                       protocol: TCP                                                                                                                                                                                                                                       
                                       to:                                                                                                                                                                                                                                                 
                                       - 0.0.0.0/0                                                                                                                                                                                                                                         
                                     - comment: allow outgoing dns via tcp                                                                                                                                                                                                                 
                                       ports:                                                                                                                                                                                                                                              
                                       - 53                                                                                                                                                                                                                                                
                                       protocol: TCP                                                                                                                                                                                                                                       
                                       to:                                                                                                                                                                                                                                                 
                                       - 0.0.0.0/0                                                                                                                                                                                                                                         
                                     - comment: allow outgoing dns and ntp via udp                                                                                                                                                                                                         
                                       ports:                                                                                                                                                                                                                                              
                                       - 53                                                                                                                                                                                                                                                
                                       - 123                                                                                                                                                                                                                                               
                                       protocol: UDP                                                                                                                                                                                                                                       
                                       to:                                                                                                                                                                                                                                                 
                                       - 0.0.0.0/0                                                                                                                                                                                                                                         
                                     ingress:                                                                                                                                                                                                                                              
                                     - comment: allow incoming ssh only to one ip                                                                                                                                                                                                          
                                       ports:                                                                                                                                                                                                                                              
                                       - 22                                                                                                                                                                                                                                                
                                       protocol: TCP
                                       from:
                                       - 0.0.0.0/0
                                       - 1.2.3.4/32
                                       to:
                                       - 212.34.83.19/32
                                     - comment: allow incoming https to all targets
                                       ports:
                                       - 80
                                       - 433
                                       protocol: TCP
                                       from:
                                       - 0.0.0.0/0

[robertvolkmann]

@majst01 forwarding is still forbidden. Do we need a firewall-controller in the mini-lab?

[majst01]

@majst01 forwarding is still forbidden. Do we need a firewall-controller in the mini-lab?

Should work with a firewall ruleset which allows it. can you check whats in the /etc/nftables/rules file ?

[robertvolkmann]

rules.txt

[majst01]

rules.txt

Looks good:

ip saddr { 10.0.0.0/8 } ip daddr 0.0.0.0/0 tcp dport { 443 } counter accept comment "allow outgoing https"
ip saddr { 10.0.0.0/8 } ip daddr 0.0.0.0/0 tcp dport { 53 } counter accept comment "allow outgoing dns via tcp"
ip saddr { 10.0.0.0/8 } ip daddr 0.0.0.0/0 udp dport { 53,123 } counter accept comment "allow outgoing dns and ntp via udp"

What does journalctl say ?

[robertvolkmann]

Jul 15 16:30:12 fw kernel: nftables-metal-dropped: IN=vrf20 OUT=vlan104009 MAC=42:61:1b:94:d5:5b:02:42:ac:11:00:04:08:00 SRC=10.0.0.2 DST=1.1.1.1 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=63395 DF PROTO=TCP SPT=46700 DPT=80 WINDOW=35840 RES=0x00 SYN URGP=0 

[robertvolkmann]

firewall-controller.service and droptailer.service are in a crash loop.

[mwindower]

The test tries to connect with port 80 which is not in the allowed rule set.

[majst01]

BTW: Ping, e.g. ICMP is actually not supported with the firewall-rules set during boot :-)

You should check if you can curl https://1.1.1.1

[robertvolkmann]

I already switched to https://1.1.1.1 but it still doesn't work on the GitHub runners.

[robertvolkmann]

Accessing the internet from the machine works on the GitHub runner as well, but SSH login to the firewall still does not work.

[majst01]

Accessing the internet from the machine works on the GitHub runner as well, but SSH login to the firewall still does not work.

This might be because there need to be a input rule added which allows ssh access

[robertvolkmann]

The firewalls have an input chain allowing SSH access:

tcp dport ssh ct state new counter accept comment "SSH incoming connections"

instead of the input chain that is present in production:

iifname "tailscale*" accept comment "Accept tailscale traffic"

It works locally without any issue but not on the GitHub runner.

[robertvolkmann]

@majst01 Thanks for remaining on: It’s always MTU, unless it’s DNS.

[Gerrit91]

LGTM