Closed sepich closed 5 years ago
neolit123
commented
6 years ago @sepich
please, create a shell script from this and show me what output do you get:
#!/bin/sh
cut -f1 -d" " /proc/modules
echo -----
version=`cut -f1 -d" " /proc/sys/kernel/osrelease`
echo $version
echo -----
cut -f1 -d" " /lib/modules/$version/modules.builtin
sepich
commented
6 years ago Please note that it is CI VM which is created, booted up, and then terminated after tests.
Then i can load modules by accessing them:
root@ip-10-99-134-230:~# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
root@ip-10-99-134-230:~# lsmod | grep ip_vs
ip_vs 147456 0
libcrc32c 16384 1 ip_vs
nf_conntrack 114688 7 ip_vs,nf_conntrack_ipv4,nf_conntrack_netlink,nf_nat_masquerade_ipv4,xt_conntrack,nf_nat_ipv4,nf_natAnd then script output changes:
It is plain debian stretch with apt install ipvsadm
neolit123
commented
6 years ago the validation code requires the following modules to be present:
ip_vs, ip_vs_rr, ip_vs_wrr, ip_vs_sh, nf_conntrack_ipv4
you have ip_vs and nf_conntrack_ipv4 but the rest are missing.
i do not understand these missing modules and IPVS in general.
the validation was added here: https://github.com/kubernetes/kubernetes/pull/59566
/cc @stewart-yu @luxas /cc @kubernetes/sig-cluster-lifecycle-bugs /cc @kubernetes/sig-node-bugs
dims
commented
6 years ago cc @m1093782566 @stewart-yu
stewart-yu
commented
6 years ago Does your cluster still using ipvs proxy, even that occur thoes warning messages? @sepich
IMO, those warning are pre-check when you use kubeadm to creating cluster. But actually, all ipvs modules will loaded by kube-proxy, please see :https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/proxier.go#L469
sepich
commented
6 years ago Does your cluster still using ipvs proxy, even that occur thoes warning messages?
Yes, i see Services in ipvsadm
those warning are pre-check when you use kubeadm to creating cluster. But actually, all ipvs modules will loaded by kube-proxy
And that is exactly reason it seems confusing - why this warning is needed then?
neolit123
commented
6 years ago And that is exactly reason it seems confusing - why this warning is needed then?
i guess a preflight warning about missing critical IPVS modules is nice to have - i.e. if required module is missing on the system. but then it becomes a question of what modules are really required?
this logic here: https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/proxier.go#L472 throws a warning by the proxier on missing modules from this list: https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/proxier.go#L157
xlgao-zju
commented
6 years ago but then it becomes a question of what modules are really required?
Agree, we should figure it out and I'd like to file a PR to fix this. :)
neolit123
commented
6 years ago i would hold on this @xlgao-zju , because even the recent blog post about IPVS included the whole list of modules that is discussed in this thread. i reviewed the blog before it was push, but to my understanding the list in there is the defacto valid list.
https://kubernetes.io/blog/2018/07/09/ipvs-based-in-cluster-load-balancing-deep-dive/ ^ @timothysc
what we don't know here is why some of the modules are not required and why it works for @sepich without them. @stewart-yu do you know why? did you see the above blog post too?
are these modules optional?:
ip_vs_rr, ip_vs_wrr, ip_vs_sh
cu
commented
6 years ago I see the same warning when joining a new node to the cluster. None of the modules are loaded into the kernel to start with, but after running kubeadm join, they are all there:
Before:
# lsmod | grep ^ip_vs
(nothing)After:
# lsmod | grep ^ip_vs
ip_vs_sh 16384 0
ip_vs_wrr 16384 0
ip_vs_rr 16384 0
ip_vs 147456 6 ip_vs_wrr,ip_vs_rr,ip_vs_shIs it a case of the preflight check not understanding that modules can be dynamically loaded when needed?
I'm running v1.11.0 from the apt repo on Ubuntu 16.04.
kvaps
commented
5 years ago Hi, I have the same problem with my 4.15.18-7-pve kernel.
Modules existing, loaded and working fine:
# modprobe -- br_netfilter ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack_ipv4
# lsmod | grep '^\(ip_vs\|ip_vs_rr\|ip_vs_wrr\|ip_vs_sh\|nf_conntrack_ipv4\)'
ip_vs_sh 16384 0
ip_vs_wrr 16384 0
ip_vs_rr 16384 16
ip_vs 151552 22 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack_ipv4 16384 7even after that I have this warning:
[preflight] running pre-flight checks
[WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack_ipv4] or no builtin kernel ipvs support: map[ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{}]
you can solve this problem with following methods:
1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs supportkube-proxy uses iptables mode by default, but if I forcefully set parameter mode: "ipvs" in it's configmap it is start using ipvs without any problems.
According this PR I found out that modules are checking inside modules.builtin file. But my one have no records for those modules:
# kernelVersion=$(cat /proc/sys/kernel/osrelease)
# cat /lib/modules/${kernelVersion}/modules.builtin | grep ipvs
<nothing>As can I see those modules existing in my modules path
# find /lib/modules/${kernelVersion} -name ip_vs.ko -o -name ip_vs_rr.ko -o -name ip_vs_wrr.ko -o -name ip_vs_sh.ko -o -name nf_conntrack_ipv4.ko
/lib/modules/4.15.18-7-pve/kernel/net/ipv4/netfilter/nf_conntrack_ipv4.ko
/lib/modules/4.15.18-7-pve/kernel/net/netfilter/ipvs/ip_vs.ko
/lib/modules/4.15.18-7-pve/kernel/net/netfilter/ipvs/ip_vs_rr.ko
/lib/modules/4.15.18-7-pve/kernel/net/netfilter/ipvs/ip_vs_sh.ko
/lib/modules/4.15.18-7-pve/kernel/net/netfilter/ipvs/ip_vs_wrr.koVerdict:
modules.builtin.kubeadm and kube-proxy
bart0sh
commented
5 years ago Please, review my attempt to fix this issue: using 'modinfo
islinwb
commented
5 years ago @kvaps
https://github.com/kubernetes/kubernetes/pull/63850 fixes https://github.com/kubernetes/kubernetes/issues/63801. The added check is just an OR operation. I suggest you to read that part of code. Where the modules are placed varies on operation systems and releases. Some are in builtin.modules and some in kernel/net/netfilter/ipvs/.
I think the kube-proxy works fine on this check.
kvaps
commented
5 years ago @islinwb, seems there is same problem, it's checking modules only inside modules.builtin file, and this check is not working in my case .
stewart-yu
commented
5 years ago @kvaps run
cut -f1 -d "" /proc/modules | grep ipvsin your machine have a try?
kvaps
commented
5 years ago @stewart-yu , sure
# cut -f1 -d " " /proc/modules | grep ip_vs
ip_vs_sh
ip_vs_wrr
ip_vs_rr
ip_vsIn word, RequiredIPVSKernelModulesAvailable() first part : https://github.com/kubernetes/kubernetes/blob/master/pkg/util/ipvs/kernelcheck_linux.go#L47-L59 will check the kernel modules
var ipvsModules = []string{
"ip_vs",
"ip_vs_rr",
"ip_vs_wrr",
"ip_vs_sh",
"nf_conntrack_ipv4",
}If missing any kerner modules, https://github.com/kubernetes/kubernetes/blob/master/pkg/util/ipvs/kernelcheck_linux.go#L62 the condition are true, we will check if exists builtin modules :https://github.com/kubernetes/kubernetes/blob/master/pkg/util/ipvs/kernelcheck_linux.go#L63-L84. If builtin modules missing, we throw warning : https://github.com/kubernetes/kubernetes/blob/master/pkg/util/ipvs/kernelcheck_linux.go#L85-L90
So, if all kernel modules in your machine, this condition in https://github.com/kubernetes/kubernetes/blob/master/pkg/util/ipvs/kernelcheck_linux.go#L62 are false, can not throwwarning
stewart-yu
commented
5 years ago So missing nf_conntrack_ipv4, please install it, and have a try @kvaps
kvaps
commented
5 years ago It's not missing:
cut -f1 -d " " /proc/modules | grep nf_conntrack_ipv4you just asked me for grep only ipvs string before
kvaps
commented
5 years ago @kvaps, Can anybody check if this PR fixes the issue?
I'm working on that
islinwb
commented
5 years ago What happened is clear:
kubeadm first checks the loaded modules;
and if ipvs modules are not all loaded, it will check the builtin modules;kube-proxy checks builtin modules;
and also loads modules, then check if they are loaded.The kubeadm doesn't automatically load these modules.
If you load these modules manually, then the kubeadm check should pass.
stewart-yu
commented
5 years ago From your waring describe in https://github.com/kubernetes/kubeadm/issues/975#issuecomment-430990149, all kernel module missing (the second line)
[preflight] running pre-flight checks
[WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack_ipv4] or no builtin kernel ipvs support: map[ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{}]
you can solve this problem with following methods:
1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs supportBut you install all kernel module first, so strange
stewart-yu
commented
5 years ago It's not missing:
cut -f1 -d " " /proc/modules | grep nf_conntrack_ipv4you just asked me for grep only
ipvsstring before
sorry, my mistakes
bart0sh
commented
5 years ago Here is my take on it: Current code of RequiredIPVSKernelModulesAvailable check doesn't check if modules are installed. It only checks if they're loaded or builtin. This confuses users as even if modules are not loaded and not builtin kube-proxy can load them if they are installed. The check forces users to manually load modules, which is not needed as kube-proxy can do it.
stewart-yu
commented
5 years ago Here is my take on it: Current code of RequiredIPVSKernelModulesAvailable check doesn't check if modules are installed. It only checks if they're loaded or builtin. This confuses users as even if modules are not loaded and not builtin kube-proxy can load them if they are installed. The check forces users to manually load modules, which is not needed as kube-proxy can do it.
yes, you are right :+1:
kvaps
commented
5 years ago @islinwb @stewart-yu My bad, I was loading modules wrong, using this command:
modprobe ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack_ipv4This is wrong, because in this case modprobe reads only first argument, and return no errors. This behavior is different unlike rmmod which removes all modules from it's arguments.
If I do:
for i in ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack_ipv4; do modprobe $i; doneall modules are loaded fine, and there is no warning from kubeadm, but after I join, kube-proxy still not uses ipvs by default:
1 server_others.go:295] Flag proxy-mode="" unknown, assuming iptables proxy
1 server_others.go:148] Using iptables Proxier.My kube-proxy config really have mode: "", if I switch it to mode: "ipvs", then ipvs is working fine. And even if modules are not loaded, it is loading them automatically.
Is that normal behavior? - In my opinion not, because in case when mode is not set it should use ipvs by default, and only if it's not accessible it should fallback to iptables, am I right?
@bart0sh I still need little bit time for testing your PR
m1093782566
commented
5 years ago @kvaps
Seems you got the key :)
Please keep in mind that ipvs mode is still not the default although it's already GA. So, you should specify the "ipvs" mode in your kube-proxy config file.
kvaps
commented
5 years ago @bart0sh I've tried your patch, it's working fine for me, no warning in kubeadm even if modules unloaded in my system. Cheers!
@m1093782566 thanks for clarification
bart0sh
commented
5 years ago @kvaps Thank you for testing it!
stewart-yu
commented
5 years ago @islinwb @stewart-yu My bad, I was loading modules wrong, using this command:
modprobe ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack_ipv4This is wrong, because in this case
modprobereads only first argument, and return no errors. This behavior is different unlikermmodwhich removes all modules from it's arguments.If I do:
for i in ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack_ipv4; do modprobe $i; doneall modules are loaded fine, and there is no warning from kubeadm, but after I join, kube-proxy still not uses ipvs by default:
1 server_others.go:295] Flag proxy-mode="" unknown, assuming iptables proxy 1 server_others.go:148] Using iptables Proxier.My kube-proxy config really have
mode: "", if I switch it tomode: "ipvs", then ipvs is working fine. And even if modules are not loaded, it is loading them automatically.Is that normal behavior? - In my opinion not, because in case when
modeis not set it should useipvsby default, and only if it's not accessible it should fallback toiptables, am I right?@bart0sh I still need little bit time for testing your PR
ipvs kube-proxy is not default model. so the behavior using iptables is normally
stewart-yu
commented
5 years ago IMO, the warning messages not confused, just throw warning, and not auto-installed modules for user. That's OK now.
If ipvs became default kube-proxy model someday, we can add code about auto-installed modules.
BTW, some disscuss in the issuse, may just related to local ENV, not about RequiredIPVSKernelModulesAvailable :slightly_smiling_face:
bart0sh
commented
5 years ago IMO, the warning messages not confused, just throw warning, and not auto-installed modules for user. That's OK now.
the warnings message says "the IPVS proxier will not be used", which is 100% incorrect. It can be used as you've just pointed out. It is confusing.
keithdadkins
commented
5 years ago FYI, I am here because of the "...IPVS proxier will not be used..." message during kubeadm join .... and it is confusing. But, wouldn't just changing the message to something like the following help clear things up?
[WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier may not be used because the following required kernel modules are not loaded: [ip_vs_sh ip_vs ip_vs_rr ip_vs_wrr] or no builtin kernel ipvs support was found: map[ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{}]. However, these modules may be loaded automatically by kube-proxy for you if they are available on your system. To verify IPVS support:
- Run
lsmod | grep 'ip_vs\|nf_conntrack'and verify each of the above modules are listed.If they are not listed, you can use the following methods to load them:
- For each missing module run 'modprobe $modulename' (e.g., 'modprobe ip_vs', 'modprobe ip_vs_rr', ...)
- If 'modprobe $modulename' returns an error, you will need to install the missing module support for your kernel. (see http://linktosomepagewithhelponinstalling.foo)
bart0sh
commented
5 years ago @keithdadkins thank you for the less confusing warning message! I updated it as you've suggested in this PR
bart0sh
commented
5 years ago /lifecycle active
What keywords did you search in kubeadm issues before filing this one?
ip_vs, RequiredIPVSKernelModulesAvailable
Is this a BUG REPORT or FEATURE REQUEST?
BUG REPORT
Versions
kubeadm version (use
kubeadm version): kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:14:41Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}Environment:
kubectl version): Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:08:34Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}uname -a): Linux debian 4.9.0-6-amd64 SMP Debian 4.9.88-1+deb9u1 (2018-05-07) x86_64 GNU/LinuxWhat happened?
ipvs is installed but modules not loaded yet, kubeadm show warning RequiredIPVSKernelModulesAvailable
But after cluster is up, ipvs modules are actually loaded by kube-proxy and i see Services VIPs in
ipvsadm -lnWhat you expected to happen?
This is confusing. Maybe warning should only be shown when such modules are not exist in system?