Closed wherka-ama closed 2 years ago
Here is the patch that I was referring to in the description:
diff --git a/pkg/cluster/internal/providers/docker/provision.go b/pkg/cluster/internal/providers/docker/provision.go
index 97b05594..aab4a3f4 100644
--- a/pkg/cluster/internal/providers/docker/provision.go
+++ b/pkg/cluster/internal/providers/docker/provision.go
@@ -286,12 +286,12 @@ func getProxyEnv(cfg *config.Cluster, networkName string, nodeNames []string) (m
noProxyList := append(subnets, envs[common.NOProxy])
noProxyList = append(noProxyList, nodeNames...)
- // Add pod and service dns names to no_proxy to allow in cluster
+ // Add pod, service and control plane(API server) dns names to no_proxy to allow in cluster
// Note: this is best effort based on the default CoreDNS spec
// https://github.com/kubernetes/dns/blob/master/docs/specification.md
// Any user created pod/service hostnames, namespaces, custom DNS services
// are expected to be no-proxied by the user explicitly.
- noProxyList = append(noProxyList, ".svc", ".svc.cluster", ".svc.cluster.local")
+ noProxyList = append(noProxyList, ".svc", ".svc.cluster", ".svc.cluster.local", strings.Join([]string{cfg.Name, "control-plane"}, "-"))
noProxyJoined := strings.Join(noProxyList, ",")
envs[common.NOProxy] = noProxyJoined
envs[strings.ToLower(common.NOProxy)] = noProxyJoined
diff --git a/pkg/cluster/internal/providers/podman/provision.go b/pkg/cluster/internal/providers/podman/provision.go
index a515324e..68c1a2a0 100644
--- a/pkg/cluster/internal/providers/podman/provision.go
+++ b/pkg/cluster/internal/providers/podman/provision.go
@@ -252,12 +252,12 @@ func getProxyEnv(cfg *config.Cluster, networkName string) (map[string]string, er
return nil, err
}
noProxyList := append(subnets, envs[common.NOProxy])
- // Add pod and service dns names to no_proxy to allow in cluster
+ // Add pod, service and control plane(API server) dns names to no_proxy to allow in cluster
// Note: this is best effort based on the default CoreDNS spec
// https://github.com/kubernetes/dns/blob/master/docs/specification.md
// Any user created pod/service hostnames, namespaces, custom DNS services
// are expected to be no-proxied by the user explicitly.
- noProxyList = append(noProxyList, ".svc", ".svc.cluster", ".svc.cluster.local")
+ noProxyList = append(noProxyList, ".svc", ".svc.cluster", ".svc.cluster.local", strings.Join([]string{cfg.Name, "con
trol-plane"}, "-"))
noProxyJoined := strings.Join(noProxyList, ",")
envs[common.NOProxy] = noProxyJoined
envs[strings.ToLower(common.NOProxy)] = noProxyJoined
I can create a PR as well if that's a preferred way of solving this problem relatively quickly.
I think we probably should add all the container names and the ~container subnet~ to the NO_PROXY
at least the node-names are available on that function
/cc @BenTheElder
EDIT
we are already passing the container subnet in the provision.go , it seems only the container names are missing
/assign @wherka-ama please go ahead
@aojea : the basic fix - just for the control plane has been added. I will check now how we can add all the container names in a similar fashion.
@aojea : the basic fix - just for the control plane has been added. I will check now how we can add all the container names in a similar fashion.
better to use an independent PR with the whole fix please, clusters with multiple control-plane will still fail with the basic fix
better to use an independent PR with the whole fix please, clusters with multiple control-plane will still fail with the basic fix
@aojea : alright, I'll push something more sophisticated then(still based on https://github.com/kubernetes-sigs/kind/pull/2885, I'll just push more/improved stuff in there). I think I know roughly how it should look like. I will test it with a multi-node/plane scenario to ensure that it does what it is supposed to. Thanks for your guidance! Much appreciated.
I've pushed an improved implementation to address the multi-node environment(@aojea :thanks for pointing that out!) and adhering to an existing MakeNodeNamer factory which will help in making the whole solution aligned with the rest of the code. I've tested it in a single and multi-node configurations with and without the proxy and it looks like we have something that works rather nicely.
See: https://github.com/kubernetes-sigs/kind/pull/2885
@aojea and @BenTheElder : do you mind having a look please?
BTW: I wrote some unit tests for that code. However, there is a little bit of a problem with mocking the exec.Command
which is buried within provision.getSubnets
. It will take some effort to make it work as a proper unit test i.e. without running the actual sys calls and without it being super ugly and incomprehensible. I'm happy to spend more time on that part, but I suggest we tackle it in the next release.
@aojea: Closing this issue.
What happened: Kind failed to create the cluster in the environment with http(s) proxy
What you expected to happen: Successful cluster creation
How to reproduce it (as minimally and precisely as possible): 1.Setup or use an existing http(s) - in this context we used squid/4.15 2.Set the env variables: https_proxy and http_proxy to point at the proxy above
Anything else we need to know?: I've done some extra troubleshooting to recreate the API server health checks done during the cluster boot:
I've already modified the podman.getProxyEnv in order to add
<cluster name>-control-plane
to no_proxy to improve the situation. After the patch everything worked as expected. I'm happy to propose such implementation as a PR.Environment:
kind version: (use
kind version
): 0.14.0Kubernetes version: (use
kubectl version
): Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.3", GitCommit:"aef86a93758dc3cb2c658dd9657ab4ad4afc21cb", GitTreeState:"clean", BuildDate:"2022-07-13T14:30:46Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.4 Unable to connect to the server: dial tcp 10.xxx:443: i/o timeoutDocker version: (use
docker info
):OS (e.g. from
/etc/os-release
):