Open justenwalker opened 4 years ago
With a newer kind version and increased -v we can get more details
I'm not actually sure what requires this other than obviously we need enough IPs for the pods
If newer versions of Kind have easier ways to troubleshoot that's great.
The issue I'm bringing up is that this subnet configuration could be validated before submission so that you get a more useful error than F0114 15:54:20.877255 1 node_ipam_controller.go:110] Controller: Invalid --cluster-cidr, mask size of cluster CIDR must be less than --node-cidr-mask-size
ah ok, that's fun. I bet kind or kubeadm neeeds to be overridding these. cc @neolit123 I vaguely remember you commenting on an issue like this
The issue I'm bringing up is that this subnet configuration could be validated before submission so that you get a more useful error than F0114 15:54:20.877255 1 node_ipam_controller.go:110] Controller: Invalid --cluster-cidr, mask size of cluster CIDR must be less than --node-cidr-mask-size
kubeadm has some special casing for the cluster CIDR mask for IPv6, but for IPv4 it does not (e.g.) take the cluster CIDR mask and increase it by one to satisfy the Node CIDR semantic of the KCM.
if you'd like kubeadm to behave differently, please log an issue in the kubernetes/kubeadm tracker.
thanks /close
@neolit123: Closing this issue.
it might be reasonable for kind do this calculation and pass it on to kubeadm, since we're providing an option for the CIDR at a level above kubeadm (and e.g. also passing it to our CNI impl).
(pass it on to kubeadm by way of patching the ---node-cidr-mask-size in the generated config)
@neolit123 / @BenTheElder It's not that I want kubernetes/kubeadmin to behave differently; I want Kind to validate the cluster config so it doesn't pass options to kubeadmin that are known to fail: ie -- providing an IPv4 CIDR that is too small.
I think it would be less confusing if kind kicked back and error that said "hey, that podSubnet you passed is way too small and the cluster wont come up" instead of kind saying "all good" and then finding out later that its in a crash-loop
it might be reasonable for kind do this calculation and pass it on to kubeadm, since we're providing an option for the CIDR at a level above kubeadm (and e.g. also passing it to our CNI impl).
it is possible for kind to manage this too.
minimal configuration:
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.16.0
networking:
podSubnet: <some-cidr>
controllerManager:
extraArgs:
node-cidr-mask-size: <mask-adapted-from-some-cidr>
@neolit123 / @BenTheElder It's not that I want kubernetes/kubeadmin to behave differently; I want Kind to validate the cluster config so it doesn't pass options to kubeadmin that are known to fail: ie -- providing an IPv4 CIDR that is too small.
using kubeadm without kind, the same failure will be observed. therefore, it is not a bad idea to have the validation on this lower level.
You can not validate this because you don't know the node-cidr-mask-size
in advance and you can not assume that the user is going to use the default value, because that is only used by the controller-manager, not cluster wide.
This is a network planning problem, because user says "my cluster has network 192.168.0.0/28" but user does not change the node-cidr-mask (that is only used by the controller-manager) , then the controller-manager uses a /24, and since it is bigger it breaks.
Cluster-CIDR is problematic, there are a lot of issues about it, but this is the one that describes it better https://github.com/kubernetes/kubernetes/issues/46508#issuecomment-709619900
scratch my last comment, lack of coffee, kubeadm is already doing the calculation, just need to be updated
@aojea I defer to you on this one. Perhaps at least we should add a docs note.
/assign
What happened:
Whe configuring a pod subnet, a subnet smaller than
/24
will cause thekube-controller-manager-kind-control-plane
to enter a CrashLoop due to this error:What you expected to happen:
If there is a minimum size of a podSubnet, the config should validate that the subnet given meets the criteria. Otherwise it should function normally (perhaps by adjusting the CIDR mask)
How to reproduce it:
Anything else we need to know?:
Environment: