Cannot use IPVS kube-proxy in strict mode

[iavael]

Summary

After switching to strict mode kubelite fails on start with error

Proxy exited can't use the IPVS proxier: error getting ipset version, error: executable file not found in $PATH

What Should Happen Instead?

kubelite should start normally

Reproduction Steps

1. Setup single-node microk8s (version 1.26)
2. Add to /var/snap/microk8s/current/args/kube-proxy these parameters:

   --proxy-mode=ipvs
   --ipvs-strict-arp

3. Make snap refresh microk8s --channel=1.26-strict/stable

Introspection Report

inspection-report-20230316_113950.tar.gz

Can you suggest a fix?

Add ipset executables (for different ipset API versions in different kernels) to microk8s snap

Are you interested in contributing with a fix?

No

[neoaggelos]

Thanks for raising this issue @iavael and apologies for missing it so far.

Indeed, for this to work in strict confinement we have to bundle ipset along with our snap. The source code for ipset can be found here https://git.netfilter.org/ipset/, and we should pick a version that would be compatible with our current iptables version.

This item is not in our team plans for the foreseeable future, but we are very eager to accept a user contribution. As a helpful reference, here is how iptables and dependencies are currently built in the snap: https://github.com/canonical/microk8s/blob/9369244ab8c94571b783dfa60c923957b6f5725e/snap/snapcraft.yaml#L126-L152

[iavael]

@neoaggelos it's a bit more complex than you described. Ipset API is independent from iptables and also less stable (it varies between different kernel versions). So it should be not one ipset binary, but multiple (for whole range of ipset API versions of supported kernels). Also some mechanism is required to choose correct binary according to kernel version in runtime. So it's not enough to build one specific version of ipset and use only it.

But I got the idea about user contribution. Maybe I would try to look at it if I'll have free time.

[neoaggelos]

Sounds good. I see that the relevant code in Kubernetes has a bunch of tests regarding kernel and ipset versions, perhaps they could help with coming up with an implementation https://github.com/kubernetes/kubernetes/blob/8f94681cd294aa8cfd3407b8191f6c70214973a4/pkg/proxy/ipvs/proxier_test.go#L284-L374

Thanks!

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.