search

cloudnativelabs / kube-router

Kube-router, a turnkey solution for Kubernetes networking.
https://kube-router.io
Apache License 2.0
2.33k stars 471 forks source link

fix(DSR): setup DSR inside pod on local eps only #1651

Closed aauren closed 7 months ago

aauren commented 7 months ago

Only attempt to setup DSR inside containers for local endpoints. Setting up DSR inside the containers network namespace requires local pods / endpoints.

Fixes #1640

@opipenbe - I tested this locally with my smaller development cluster and it appeared to work correctly, however, I would be very appreciative if you would test this in your cluster as well. A container should be built as part of the GitHub Actions CI on this PR, you can look in the CI log for a link to kube-router-git for the newly built kube-router.

One thing that I still don't quite have an answer for though, is why this didn't also have this same error for you under kube-router v1.6.0 as the code path was essentially the same: https://github.com/cloudnativelabs/kube-router/blob/v1.6/pkg/controllers/proxy/service_endpoints_sync.go#L444-L447

Maybe something further up in the calling tree filtered it out?

Anyway, since to me, this seems like it is changing DSR logic that has existed for a long time, I'd appreciate the second test to ensure that it not only removes the error, but also still allows you to route to your service correctly.

opipenbe commented 7 months ago

Thank you @aauren for the great work!

I just tested with PR-1651 image in my environment and I can confirm DSR feature now works correctly! I also did full environment restart in order to be sure this change does fix DSR.

aauren commented 7 months ago

@opipenbe - Thanks for testing! Just to be double-sure, you also tested that the service behind the DSR also still continued to work in different scenarios too right? (e.g. from a worker that didn't contain the endpoint, for outside the cluster, etc.)?

opipenbe commented 7 months ago

@aauren , I tested DSR outside of the cluster since my service external IP is exposed from all worker nodes by BGP.

I also didnt find anything suspicious in the kube-router logs.

I dont know a good method to test DSR inside cluster. As I understand, it is by design that packets are dropped from cluster to external IPs with DSR feature? If It is somehow possible to test DSR inside the cluster, then let me know and I can test it also :)

aauren commented 7 months ago

Hmm... In my cluster, the external IPs route within the cluster just fine DSR or no DSR.

Inside the cluster, for non-service-local VIPs every node should have an IPVS definition for the service, so I would imagine that it should "just work".

opipenbe commented 7 months ago

Thanks @aauren for clarification! I figured out why DSR packets are dropped in my environment - in the RHEL / Rocky Linux, multihoming is not enabled by default. DSR makes routing asymmetric, so rp_filter (default value 1) will drop DSR: https://access.redhat.com/solutions/53031 .

So I can also confirm that DSR is working successfully inside cluster after configuring kernel parameter rp_filter = 2 for all nodes :)

aauren commented 7 months ago

@opipenbe Thanks for taking the deep dive on this!

I have added a commit to this, that sets rp_filter to 2 during the startup of kube-router similar to other sysctl values that we do sanity checks on.