Allow replace kube-proxy by kube-router Service Proxy

[leleobhz]

Is your feature request related to a problem? Please describe.

In Lightweight environments, kube-proxy may be replaced by kube-router service proxy (--run-service-proxy). This mode uses IPVS (And this way, it's directly compatible with MetalLB - even more if flag --advertise-loadbalancer-ip got added to ClusterConfig) and uses just one service to provide networking.

Describe the solution you would like

No response

Describe alternatives you've considered

No response

Additional context

No response

[twz123]

There's a related discussion on the forums. The gist of it: You can try to configure k0s in this way, by disabling kube-proxy (--disable-components=kube-proxy) and deploying kube-router as a custom CNI.

Note that in the upcoming k0s 1.29 release, there will be the ability to pass extra arguments to the k0s-managed kube-router deployment. You may be able to get by without deploying kube-router yourself by adding --run-service-proxy to the kube-router extraArgs section of the k0s config. See #3902 for details.

[leleobhz]

There's a related discussion on the forums. The gist of it: You can try to configure k0s in this way, by disabling kube-proxy (--disable-components=kube-proxy) and deploying kube-router as a custom CNI.

Note that in the upcoming k0s 1.29 release, there will be the ability to pass extra arguments to the k0s-managed kube-router deployment. You may be able to get by without deploying kube-router yourself by adding --run-service-proxy to the kube-router extraArgs section of the k0s config. See #3902 for details.

Hello!

I tried a variant of this, editing daemonset to add --run-service-proxy by my own and disabling kubeproxy on spec.network. But for some reason, this did not worked as expected because kube-router did not configured ipvs as expected. Also, I see k0s uses 1.x kube-proxy version instead 2.x and I dont know if this changes something.

About #3902 - this commit is not expected to be released on 1.28 series?

[leleobhz]

Also, there is any issue of using kube-router for service-proxy as default if user disables kube-proxy? There is some advantage to keep kube-proxy instead using by default kube-router service-proxy suppport?

[twz123]

Also, I see k0s uses 1.x kube-proxy version instead 2.x and I dont know if this changes something.

I guess you mean kube-router, not kube-proxy? The update is being worked on (#3814).

About #3902 - this commit is not expected to be released on 1.28 series?

This is not planned. The 1.29 release will not take too long anymore.

Also, there is any issue of using kube-router for service-proxy as default if user disables kube-proxy? There is some advantage to keep kube-proxy instead using by default kube-router service-proxy suppport?

I'll refer to @juanluisvaladas's answer: http://forums.k8slens.dev/t/how-to-enable-kube-routers-load-balance-feature-under-k0s/800/4

[jnummelin]

1.29 release will have new feature which allows you to configure any args for kube-router. With that you can the enable proxy in kube-router and disable k0s setting up kube-proxy.

[leleobhz]

To keep the scope on track, I suggest we close this issue since everything is tracked here.

But a question that @juanluisvaladas awnser got me: If kube-proxy implements LoadBalancer, why this is not mentioned in https://docs.k0sproject.io/v1.28.5+k0s.0/examples/nginx-ingress/?h=loadbalance#install-nginx-using-loadbalancer as example?

In background (But will not ask this here now) i'm investigating a misbehave of any IPVS implementation with MetalLB (SYN goes, SYN+ACK not and I see returning packets does not have LB IP as source, but POD IP) and i'm thinking now if use kube-proxy as LB class may fix this. In this case, kube-proxy LB capability does not deserve any additional documentation?

[juanluisvaladas]

Hi @leleobhz,

But a question that @juanluisvaladas awnser got me: If kube-proxy implements LoadBalancer, why this is not mentioned in https://docs.k0sproject.io/v1.28.5+k0s.0/examples/nginx-ingress/?h=loadbalance#install-nginx-using-loadbalancer as example?

Neither kube-proxy or kube-router implement LoadBalancer type, I did say that, but it must have been a lapse because LoadBalancer is intended for external load balancers and kube-proxy was never supposed to implement it. It was 6 months ago so I'm not sure what was the missing feature I was mentioning but I remember there were a couple...

Anyway, what I said is we're willing to study feature requests as long as they are justified, the previous request didn't say why we should implement but this but you are providing some reasons to do so. So it's definitely worth considering this again.

Do you think that, given that you can disable kube-proxy and starting 1.29 you'll be able to add the flag this is still needed? Or having that suffices?

As for the kube-proxy version, @twz123 mentioned you may mean kube-router, but if you are not please clarify exactly what are you referring to, I'm aware of kpng and KEP-2104 but I don't think you're referring to that...

[leleobhz]

Hi @leleobhz,

Hi @juanluisvaladas

Anyway, what I said is we're willing to study feature requests as long as they are justified, the previous request didn't say why we should implement but this but you are providing some reasons to do so. So it's definitely worth considering this again.

I'm thankful by your consideration about this. I'm challenging myself run k0s on Rpi3 cluster because 1) k0s is the most upstream-compliant mini-distro of kubernetes I've found 2) I want to study how Kubernetes can be improved on low memory scenarios (And I got good result with NodeSwap + zram + more disk swap) 3) Learn more about Kubernetes internal. Also, Kubernetes on low requisite scenarios is a important study.

Do you think that, given that you can disable kube-proxy and starting 1.29 you'll be able to add the flag this is still needed? Or having that suffices?

I'll attempt this after 1.29 release. I'm stuck on tricky network issue here (No matter what, anything using LoadBalancer class starts 3 way handshake but after 3rd ack awnser, communication stops and I receive a tons of retransmissions until timeout) but if I can reach this point using only Kuberouter, I think it's possible consider it works. Also, I'll try change distro/kernel to check for some heavy restriction on environment (DietPI is a lot compact but may be missing something).

As for the kube-proxy version, @twz123 mentioned you may mean kube-router, but if you are not please clarify exactly what are you referring to, I'm aware of kpng and https://github.com/kubernetes/enhancements/issues/2104 but I don't think you're referring to that...

Indeed I mean kube-router version: https://github.com/k0sproject/k0s/blob/2a8c296c9121752c7f30a047eb6ec8597762819b/pkg/constant/constant_shared.go#L91

I think if Kube-router got updated to 2.x line, maybe possible to use only kube-router + metallb with kube-router provinding pod networking + IPVS proxy + BGP and metallb providing LB Class (kube-router manages BGP and with the bgp-announce-lb-ip as true it manages lb address announcement too). And on implementations that does not rely on external BGP, kuberouter can provide pod networking + IPVS proxy and metallb with LB Class plus ARP listener.

[pedro-n-rocha]

For runnning with only standalone kuberouter, kuberouter daemonset needs a proper configured kubeconfig Arg ,

this is not implemented in

https://github.com/k0sproject/k0s/blob/f29519bc3c9a8c09236f8384d5181eaa619d5764/pkg/component/controller/kuberouter.go#L183

like it it is for kubeproxy ( cleaner > server: {{ .ControlPlaneEndpoint }} )

https://github.com/k0sproject/k0s/blob/33b4f8ad4952b137236e854c494a80634425d8c6/pkg/component/controller/kubeproxy.go#L265

so only got it working with custom network provider option,and with the hardcoded internal api server IP and port 6443 on the manifest kubeconfig, ( hacky at best )

still not convinced of the benefits of disabling kube-proxy.

[leleobhz]

still not convinced of the benefits of disabling kube-proxy.

Hello @pedro-n-rocha

Main reason is system resource waist. Calico replaces kube-proxy in eBPF mode and kube-router also does support replace kube-proxy. Not telling here kube-proxy is a bad implementation, but maybe redundant in some scenarios. It's at least a good look into flexibility since k0s can be customized/extended with helm and Calico - as example - can be deployed and configured entirelly from helm charts.

[jnummelin]

@leleobhz Now that k0s supports adding the needed arg(s) to kube-router and kube-proxy can be disabled, I don't think there's anything more for k0s to do, right? If so, pls close the issue

[leleobhz]

Hi @jnummelin

I've tried this in a older version of k0s. I'll try to replicate this on 1.31 and I'll feedback here.