metallb seems to work, but can't curl to assigned IP

[nderraugh]

Hi, quick shout out to the micrk8s team: I'm super excited about microk8s. It is great, thank you!

I am setting up metallb, and running into a problem. Everything seems to be working as expected from metallb (arping, tcpdump and the speaker logs) but I'm not able to curl a connection to the metallb assigned IP from my mac on the same network, but I am from the microk8s node itself. I don't want to send anyone down an unproductive rabbit hole -- caveat lector -- but my current intuition is that it's related to iptables.

I have a 1 node microk8s instance on a raspberry pi 4b running ubuntu 20.04. Before I installed micrk8s I added cgroup_enable=memory cgroup_memory=1 to /boot/firmware/cmdline.txt and rebooted and I ran sudo iptables -P FORWARD ACCEPT. I installed microk8s this morning with snap install microk8s --classic. I have the pi plugged into my router with ethernet. I configured my router to give out IPs from 192.168.12.0/25 and reserved 192.168.12.128/25 for metallb. That's the range I used when I enabled the metallb addon:

ubuntu@ubuntu:~$ microk8s enable metallb
Enabling MetalLB
Enter each IP address range delimited by comma (e.g. '10.64.140.43-10.64.140.49,192.168.0.105-192.168.0.111'): 192.168.12.128/25

Following the verification steps from https://opensource.com/article/20/7/homelab-metallb I created a nginx landing page deployment with:

cat <<EOF | microk8s kubectl create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-verify
  namespace: kube-verify
  labels:
    app: kube-verify
spec:
  replicas: 3
  selector:
    matchLabels:
      app: kube-verify
  template:
    metadata:
      labels:
        app: kube-verify
    spec:
      containers:
      - name: nginx
        image: quay.io/clcollins/kube-verify:01
        ports:
        - containerPort: 8080
EOF

And created a service in microk8s to use a metallb LoadBalancer with:

cat <<EOF | microk8s kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: kube-verify
  namespace: kube-verify
spec:
  selector:
    app: kube-verify
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
  type: LoadBalancer
EOF

Looking at the service it assigned 192.168.12.128 as the external IP. If I ssh into the microk8s node which has an IP of 192.168.12.64 I am able to curl 192.168.12.128 and get the landing page. But if I curl from any other hosts outside the "cluster" I get no response.

From my mac if I arping I get:

➜  ~ sudo arping -I en0 192.168.12.128
ARPING 192.168.12.128
60 bytes from dc:a6:32:c3:b5:3d (192.168.12.128): index=0 time=10.402 msec
60 bytes from dc:a6:32:c3:b5:3d (192.168.12.128): index=1 time=4.061 msec
60 bytes from dc:a6:32:c3:b5:3d (192.168.12.128): index=2 time=4.491 msec
60 bytes from dc:a6:32:c3:b5:3d (192.168.12.128): index=3 time=12.808 msec
60 bytes from dc:a6:32:c3:b5:3d (192.168.12.128): index=4 time=4.096 msec
60 bytes from dc:a6:32:c3:b5:3d (192.168.12.128): index=5 time=4.827 msec
60 bytes from dc:a6:32:c3:b5:3d (192.168.12.128): index=6 time=11.741 msec
60 bytes from dc:a6:32:c3:b5:3d (192.168.12.128): index=7 time=11.264 msec
60 bytes from dc:a6:32:c3:b5:3d (192.168.12.128): index=8 time=13.275 msec
60 bytes from dc:a6:32:c3:b5:3d (192.168.12.128): index=9 time=10.974 msec
60 bytes from dc:a6:32:c3:b5:3d (192.168.12.128): index=10 time=11.177 msec
^C
--- 192.168.12.128 statistics ---
11 packets transmitted, 11 packets received,   0% unanswered (0 extra)
rtt min/avg/max/std-dev = 4.061/9.011/13.275/3.595 ms

and from the same mac if I run tcpdump I get:

➜  ~ sudo tcpdump -n -i en0 arp src host 192.168.12.128
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on en0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:56:40.426788 ARP, Reply 192.168.12.128 is-at dc:a6:32:c3:b5:3d, length 46
11:56:41.421084 ARP, Reply 192.168.12.128 is-at dc:a6:32:c3:b5:3d, length 46
11:56:42.421840 ARP, Reply 192.168.12.128 is-at dc:a6:32:c3:b5:3d, length 46
11:56:43.430597 ARP, Reply 192.168.12.128 is-at dc:a6:32:c3:b5:3d, length 46
11:56:44.422194 ARP, Reply 192.168.12.128 is-at dc:a6:32:c3:b5:3d, length 46
11:56:45.423445 ARP, Reply 192.168.12.128 is-at dc:a6:32:c3:b5:3d, length 46
11:56:46.430586 ARP, Reply 192.168.12.128 is-at dc:a6:32:c3:b5:3d, length 46
11:56:47.430423 ARP, Reply 192.168.12.128 is-at dc:a6:32:c3:b5:3d, length 46
11:56:48.432679 ARP, Reply 192.168.12.128 is-at dc:a6:32:c3:b5:3d, length 46
11:56:49.432113 ARP, Reply 192.168.12.128 is-at dc:a6:32:c3:b5:3d, length 46
11:56:50.436955 ARP, Reply 192.168.12.128 is-at dc:a6:32:c3:b5:3d, length 46
^C
11 packets captured
10483 packets received by filter
0 packets dropped by kernel

In both the arping and tcpdump output the MAC is correct. So I'm pretty sure metallb is setup and doing what it is supposd to.

Attached is my microk8s inspect tarball.

I am happy to answer any questions, or provide more info, or try other things to try and help out. I've done this start to finish several times in the last few days and I'm pretty confident it's reproducible as per ^^, but don't have a 2nd pi handy to find out. inspection-report-20201014_160209.tar.gz

[chris-sanders]

I'm running a cluster with metallb, and the fact that your ping is returning the correct value I agree means you're just looking at networking to get connected.

My first reaction is that it looks like a firewall is blocking port 80. I don't have a pi with 20.04 around so I don't know the defaults. Is ufw on and enabled on the pi by chance? The ping would connect and the curl timeout if it was blocked by a firewall.

[nderraugh]

Hmm, thanks for jumping in with your thoughts @chris-sanders. No ufw:

ubuntu@ubuntu:~$ sudo ufw status
Status: inactive

The iptables output is in the attached inspection-report above.

[nderraugh]

The problem was a user error: misconfigured router.

[fidesachates]

https://xkcd.com/979/