k3s-created cni0 overrides nginx port rules

[oxy]

Describe the bug When k3s is started, nginx stops serving data, and instead a mysterious 404 page not found is served on all nginx domains. On using lsof -p 80, only nginx's workers appear, but even after stopping nginx, the strange 404 page persists until reboot. Then, it disappears, and nginx works again until k3s is started.

This appears to be due to https://github.com/rancher/k3s/blob/4af06882b67cd3f56285b6a5a1388b7190b93533/pkg/agent/flannel/setup.go#L26 but I'm just making a guess, as I'm new to k3s and networking.

To Reproduce Start k3s when nginx is running. Visit any domain served by nginx.

Expected behavior nginx continues to serve requests.

[nberlee]

The response you are getting is from the traefik loadbalancer.

Its a project called klipper-lb which makes nat rules (nat rules are processed before a normal port binding like your nginx). And is installed by default in k3s.

see:

kubectl logs -n kube-system -l app=svclb-traefik -c lb-port-80

This behavior is actually in the README.md of k3s:

k3s includes a basic service load balancer that uses available host ports. If you try to create a load balancer that listens on port 80, for example, it will try to find a free host in the cluster for port 80. If no port is available the load balancer will stay in Pending.

To disable the embedded service load balancer (if you wish to use a different implementation like MetalLB) just add --no-deploy=servicelb to the server on startup.

[deniseschannon]

@Oxylibrium if you have more questions, please let us know!

[r4ym0n]

I got in the same problem.And I found out it was caused by treafik's own LB. traefik LoadBalancer 10.43.127.25 192.168.66.100 80:32635/TCP,443:31959/TCP 139m use edit command to set the 80 port to 8080. to avoid overwrite situations. kubectl edit service/traefik -n kube-system