Closed asierzd closed 8 months ago
cc @chaosi-zju for help
cc @chaosi-zju for help
OK,let me try~
@asierzd Hello, it's my honor to help you to solve this problem.
Firstly, can you add this steps to your previous operation for a trial and provide the result info to me?
1、*** LOCAL:
$ kind create cluster --name localcluster --config kind-config.yaml
after that, execute:
$ docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "localcluster-control-plane"
$ kubectl config set-cluster "kind-localcluster" --server="https://xx.xx.xx.xx:6443"
inside,
xx.xx.xx.xx
is output of the previousdocker inspect
command
2、*** REMOTE:
$ kind create cluster --name remotecluster --config kind-config.yaml
after that, execute:
$ docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "remotecluster-control-plane"
$ kubectl config set-cluster "kind-remotecluster" --server="https://xx.xx.xx.xx:6443"
inside,
xx.xx.xx.xx
is output of the previousdocker inspect
command
feel free to provide feedback on any unclear areas
In addition, can you provider the logs of karmada-controller-manager
you can do like this:
# First, check the pod name of karmada-controller-manager
kubectl --kubeconfig=${kubeconfig of host cluster} get po -o wide -n karmada-system
# Then, check the logs of karmada-controller-manager
kubectl --kubeconfig=${kubeconfig of host cluster} logs -f ${pod name of karmada-controller-manager} -n karmada-system
Hello @chaosi-zju thank you very much for your help.
After performing the config set-cluster for each cluster, I can do the ssh tunnels correctly, but I can't access any of the clusters, for example in local I obtain a timeout, and therefore I can't execute the Karmada join:
*** LOCAL:
$ kubectl get nodes -v 9
I0706 09:35:32.849333 6259 loader.go:373] Config loaded from file: /home/user/.kube/config
I0706 09:35:32.850049 6259 round_trippers.go:466] curl -v -XGET -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.26.0 (linux/amd64) kubernetes/b46a3f8" 'https://172.18.0.2:6443/api?timeout=32s'
I0706 09:36:02.852129 6259 round_trippers.go:508] HTTP Trace: Dial to tcp:172.18.0.2:6443 failed: dial tcp 172.18.0.2:6443: i/o timeout
I0706 09:36:02.852200 6259 round_trippers.go:553] GET https://172.18.0.2:6443/api?timeout=32s in 30002 milliseconds
I0706 09:36:02.852243 6259 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 30002 ms TLSHandshake 0 ms Duration 30002 ms
I0706 09:36:02.852310 6259 round_trippers.go:577] Response Headers:
E0706 09:36:02.852466 6259 memcache.go:238] couldn't get current server API group list: Get "https://172.18.0.2:6443/api?timeout=32s": dial tcp 172.18.0.2:6443: i/o timeout
*** REMOTE:
# kubectl karmada --kubeconfig /etc/karmada/karmada-apiserver.config join kind-localcluster --cluster-kubeconfig=/home/ec2-user/.kube/config-local -v 9
I0706 07:44:09.123786 10889 join.go:137] Joining cluster. cluster name: kind-localcluster
I0706 07:44:09.123840 10889 join.go:138] Joining cluster. cluster namespace: karmada-cluster
I0706 07:44:09.124580 10889 loader.go:373] Config loaded from file: /etc/karmada/karmada-apiserver.config
I0706 07:44:09.125311 10889 loader.go:373] Config loaded from file: /home/ec2-user/.kube/config-local
I0706 07:44:09.126367 10889 join.go:162] Joining cluster config. endpoint: http://127.0.0.1:5252
I0706 07:44:09.126440 10889 round_trippers.go:466] curl -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl-karmada/v0.0.0 (linux/amd64) kubernetes/$Format" 'http://127.0.0.1:5252/api/v1/namespaces/kube-system'
I0706 07:44:09.127533 10889 round_trippers.go:510] HTTP Trace: Dial to tcp:127.0.0.1:5252 succeed
I0706 07:44:39.280316 10889 round_trippers.go:553] GET http://127.0.0.1:5252/api/v1/namespaces/kube-system 500 Internal Server Error in 30153 milliseconds
I0706 07:44:39.280355 10889 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 0 ms TLSHandshake 0 ms ServerProcessing 30151 ms Duration 30153 ms
I0706 07:44:39.280370 10889 round_trippers.go:577] Response Headers:
I0706 07:44:39.280381 10889 round_trippers.go:580] Content-Type: text/plain; charset=utf-8
I0706 07:44:39.280390 10889 round_trippers.go:580] X-Content-Type-Options: nosniff
I0706 07:44:39.280397 10889 round_trippers.go:580] Date: Thu, 06 Jul 2023 07:44:33 GMT
I0706 07:44:39.280404 10889 round_trippers.go:580] Content-Length: 38
I0706 07:44:39.280487 10889 request.go:1171] Response Body: dial tcp 172.18.0.2:6443: i/o timeout
I0706 07:44:39.280724 10889 helpers.go:246] server response object: [{
"metadata": {},
"status": "Failure",
"message": "an error on the server (\"dial tcp 172.18.0.2:6443: i/o timeout\") has prevented the request from succeeding (get namespaces kube-system)",
"reason": "InternalError",
"details": {
"name": "kube-system",
"kind": "namespaces",
"causes": [
{
"reason": "UnexpectedServerResponse",
"message": "dial tcp 172.18.0.2:6443: i/o timeout"
}
]
},
"code": 500
}]
Error from server (InternalError): an error on the server ("dial tcp 172.18.0.2:6443: i/o timeout") has prevented the request from succeeding (get namespaces kube-system)
Here I provide you the logs of karmada-controller-manager after the trial done with "set-cluster", in txt format: LogsKarmadaControllerAfterSetCluster.txt
Here I provide you the logs of karmada-controller-manager when I could perform the Karmada join at the beginning, in txt format: LogsKarmadaControllerAfterJoin.txt
By the way, I have discovered this error in the second log for the Join:
Failed to do cluster health check for cluster kind-localcluster, err is : Get "http://127.0.0.1:5252/readyz": dial tcp 127.0.0.1:5252: connect: connection refused
It could be related to the ip or port used for the clusters.
Let me know if you discover the problem.
Thank you very much.
I have tried again changing the apiserver ip and port as specified in another local machine and now it returns outputs of kubectl as expected (The issue with this must have been with WSL2 Ubuntu22, which is where I was testing the local cluster).
I have executed again the same steps specified at the beginning of this issue, it says it has been joined succesfully, but when I execute the kubectl --kubeconfig /etc/karmada/karmada-apiserver.config get clusters
, the Ready state is in False, and when I describe the cluster it says that the cluster is not reachable.
@asierzd According to your description, I understand that the network between the remote kind cluster container and the local cluster is disconnected. I understand that the tunnel you opened can only interwork between the port of the local and remote machines, but the Karmada component accesses the member cluster in the kind container. So, I understand you can check the remote container to the local cluster network again, hopefully to help you
=>remote cluster
docker ps -a | grep remote-control-plane
docker exec containerdID curl http://127.0.0.1:5252 -H "Bear xxxxx" -k
Container in direct access to port 5252, I understand must be network unreachable
Hello @Fish-pro, thank you for your help.
I have executed your commands with the remote cluster, and this is the result:
# docker exec aea1d6f123c6 curl http://127.0.0.1:5252 -H "Bear xxxxx" -k
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (7) Failed to connect to 127.0.0.1 port 5252: Connection refused
If I execute the curl request from the remote EC2 outside the docker container of the remote cluster, it is able to reach the api-server of the local cluster:
# curl -k http://127.0.0.1:5252 -H "Bear xxxxx"
{
"paths": [
"/.well-known/openid-configuration",
"/api",
"/api/v1",
"/apis",
"/apis/",
"/apis/admissionregistration.k8s.io",
"/apis/admissionregistration.k8s.io/v1",
"/apis/apiextensions.k8s.io",
"/apis/apiextensions.k8s.io/v1",
"/apis/apiregistration.k8s.io",
"/apis/apiregistration.k8s.io/v1",
"/apis/apps",
"/apis/apps/v1",
"/apis/authentication.k8s.io",
"/apis/authentication.k8s.io/v1",
"/apis/authorization.k8s.io",
"/apis/authorization.k8s.io/v1",
"/apis/autoscaling",
"/apis/autoscaling/v1",
"/apis/autoscaling/v2",
"/apis/batch",
"/apis/batch/v1",
"/apis/certificates.k8s.io",
"/apis/certificates.k8s.io/v1",
"/apis/coordination.k8s.io",
"/apis/coordination.k8s.io/v1",
"/apis/discovery.k8s.io",
"/apis/discovery.k8s.io/v1",
"/apis/events.k8s.io",
"/apis/events.k8s.io/v1",
"/apis/flowcontrol.apiserver.k8s.io",
"/apis/flowcontrol.apiserver.k8s.io/v1beta2",
"/apis/flowcontrol.apiserver.k8s.io/v1beta3",
"/apis/networking.k8s.io",
"/apis/networking.k8s.io/v1",
"/apis/node.k8s.io",
"/apis/node.k8s.io/v1",
"/apis/policy",
"/apis/policy/v1",
"/apis/rbac.authorization.k8s.io",
"/apis/rbac.authorization.k8s.io/v1",
"/apis/scheduling.k8s.io",
"/apis/scheduling.k8s.io/v1",
"/apis/storage.k8s.io",
"/apis/storage.k8s.io/v1",
"/healthz",
"/healthz/autoregister-completion",
"/healthz/etcd",
"/healthz/log",
"/healthz/ping",
"/healthz/poststarthook/aggregator-reload-proxy-client-cert",
"/healthz/poststarthook/apiservice-discovery-controller",
"/healthz/poststarthook/apiservice-openapi-controller",
"/healthz/poststarthook/apiservice-openapiv3-controller",
"/healthz/poststarthook/apiservice-registration-controller",
"/healthz/poststarthook/apiservice-status-available-controller",
"/healthz/poststarthook/bootstrap-controller",
"/healthz/poststarthook/crd-informer-synced",
"/healthz/poststarthook/generic-apiserver-start-informers",
"/healthz/poststarthook/kube-apiserver-autoregistration",
"/healthz/poststarthook/priority-and-fairness-config-consumer",
"/healthz/poststarthook/priority-and-fairness-config-producer",
"/healthz/poststarthook/priority-and-fairness-filter",
"/healthz/poststarthook/rbac/bootstrap-roles",
"/healthz/poststarthook/scheduling/bootstrap-system-priority-classes",
"/healthz/poststarthook/start-apiextensions-controllers",
"/healthz/poststarthook/start-apiextensions-informers",
"/healthz/poststarthook/start-cluster-authentication-info-controller",
"/healthz/poststarthook/start-deprecated-kube-apiserver-identity-lease-garbage-collector",
"/healthz/poststarthook/start-kube-aggregator-informers",
"/healthz/poststarthook/start-kube-apiserver-admission-initializer",
"/healthz/poststarthook/start-kube-apiserver-identity-lease-controller",
"/healthz/poststarthook/start-kube-apiserver-identity-lease-garbage-collector",
"/healthz/poststarthook/start-legacy-token-tracking-controller",
"/healthz/poststarthook/start-system-namespaces-controller",
"/healthz/poststarthook/storage-object-count-tracker-hook",
"/livez",
"/livez/autoregister-completion",
"/livez/etcd",
"/livez/log",
"/livez/ping",
"/livez/poststarthook/aggregator-reload-proxy-client-cert",
"/livez/poststarthook/apiservice-discovery-controller",
"/livez/poststarthook/apiservice-openapi-controller",
"/livez/poststarthook/apiservice-openapiv3-controller",
"/livez/poststarthook/apiservice-registration-controller",
"/livez/poststarthook/apiservice-status-available-controller",
"/livez/poststarthook/bootstrap-controller",
"/livez/poststarthook/crd-informer-synced",
"/livez/poststarthook/generic-apiserver-start-informers",
"/livez/poststarthook/kube-apiserver-autoregistration",
"/livez/poststarthook/priority-and-fairness-config-consumer",
"/livez/poststarthook/priority-and-fairness-config-producer",
"/livez/poststarthook/priority-and-fairness-filter",
"/livez/poststarthook/rbac/bootstrap-roles",
"/livez/poststarthook/scheduling/bootstrap-system-priority-classes",
"/livez/poststarthook/start-apiextensions-controllers",
"/livez/poststarthook/start-apiextensions-informers",
"/livez/poststarthook/start-cluster-authentication-info-controller",
"/livez/poststarthook/start-deprecated-kube-apiserver-identity-lease-garbage-collector",
"/livez/poststarthook/start-kube-aggregator-informers",
"/livez/poststarthook/start-kube-apiserver-admission-initializer",
"/livez/poststarthook/start-kube-apiserver-identity-lease-controller",
"/livez/poststarthook/start-kube-apiserver-identity-lease-garbage-collector",
"/livez/poststarthook/start-legacy-token-tracking-controller",
"/livez/poststarthook/start-system-namespaces-controller",
"/livez/poststarthook/storage-object-count-tracker-hook",
"/logs",
"/metrics",
"/metrics/slis",
"/openapi/v2",
"/openapi/v3",
"/openapi/v3/",
"/openid/v1/jwks",
"/readyz",
"/readyz/autoregister-completion",
"/readyz/etcd",
"/readyz/etcd-readiness",
"/readyz/informer-sync",
"/readyz/log",
"/readyz/ping",
"/readyz/poststarthook/aggregator-reload-proxy-client-cert",
"/readyz/poststarthook/apiservice-discovery-controller",
"/readyz/poststarthook/apiservice-openapi-controller",
"/readyz/poststarthook/apiservice-openapiv3-controller",
"/readyz/poststarthook/apiservice-registration-controller",
"/readyz/poststarthook/apiservice-status-available-controller",
"/readyz/poststarthook/bootstrap-controller",
"/readyz/poststarthook/crd-informer-synced",
"/readyz/poststarthook/generic-apiserver-start-informers",
"/readyz/poststarthook/kube-apiserver-autoregistration",
"/readyz/poststarthook/priority-and-fairness-config-consumer",
"/readyz/poststarthook/priority-and-fairness-config-producer",
"/readyz/poststarthook/priority-and-fairness-filter",
"/readyz/poststarthook/rbac/bootstrap-roles",
"/readyz/poststarthook/scheduling/bootstrap-system-priority-classes",
"/readyz/poststarthook/start-apiextensions-controllers",
"/readyz/poststarthook/start-apiextensions-informers",
"/readyz/poststarthook/start-cluster-authentication-info-controller",
"/readyz/poststarthook/start-deprecated-kube-apiserver-identity-lease-garbage-collector",
"/readyz/poststarthook/start-kube-aggregator-informers",
"/readyz/poststarthook/start-kube-apiserver-admission-initializer",
"/readyz/poststarthook/start-kube-apiserver-identity-lease-controller",
"/readyz/poststarthook/start-kube-apiserver-identity-lease-garbage-collector",
"/readyz/poststarthook/start-legacy-token-tracking-controller",
"/readyz/poststarthook/start-system-namespaces-controller",
"/readyz/poststarthook/storage-object-count-tracker-hook",
"/readyz/shutdown",
"/version"
]
I have also seen the same behavior doing the requests from the local cluster and the local machine to the remote cluster.
Could this connection refused be something related to the permissions of the Kubernetes clusters or Karmada, or is it a problem of the network with the tunnelling?
Thank you very much.
@asierzd
From this phenomenon, the tunnel you open is only between hosts, but the container network environment and the host network environment are isolated, which is the reason for the access failure. I don't quite understand why tunneling is necessary, and I understand that using 127.0.0.1
as the cluster api-server address is not a common behavior. If you just need to verify karmada's capabilities, can you modify kind-config so that the address of the api-server is remotely accessible
kind: Cluster
apiVersion: "kind.x-k8s.io/v1alpha4"
networking:
apiServerAddress: "0.0.0.0"
nodes:
- role: control-plane
@Fish-pro
I am still getting connection refused from inside the cluster after changing the cluster settings to use apiServerAddress: "0.0.0.0"
.
I am not a very advanced user of K8s, which solution would you propose to connect 2 distant/remote Kind clusters using Karmada? Maybe to use a NodePort service instead of using kubectl proxy
and/or an alternative to the ssh tunneling?
Thank you very much.
@Fish-pro I am still getting connection refused from inside the cluster after changing the cluster settings to use
apiServerAddress: "0.0.0.0"
. I am not a very advanced user of K8s, which solution would you propose to connect 2 distant/remote Kind clusters using Karmada? Maybe to use a NodePort service instead of usingkubectl proxy
and/or an alternative to the ssh tunneling?Thank you very much.
@asierzd
I described above the use of 0.0.0.0
mode, if the kind cluster deployment, the machine IP can be used as the kube api-server address, can be accessed by external, you need to change the api-server address to the machine IP
kind: Cluster
apiVersion: "kind.x-k8s.io/v1alpha4"
networking:
apiServerAddress: "0.0.0.0"
nodes:
- role: control-plane
Follow-up problems only need to ensure that the cluster where karmada resides can access this ip address
Regarding the latter two questions, I understand that if karmada is running in a kubernetes cluster, if karmada is running in a node and the local cluster is tunnelled, then I understand that the above approach is accessible in this case. However, it will bring new problems, if the karmada pod drifts to a new node, then the new node needs to ensure that the node tunnel is opened.
Hello @Fish-pro, I'm sorry for the delay, I had some trouble setting up this configuration and with other projects at work.
In order to be able to do curl requests to the api-server I had to create the cluster with this kind-config:
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
apiServerAddress: "0.0.0.0"
apiServerPort: 6443
nodes:
- role: control-plane
kubeadmConfigPatches:
- |
kind: ClusterConfiguration
apiServer:
certSANs:
- "new-context"
And then change the api-server to the IP of the control plane docker container:
container_ip=$(docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "${CLUSTER_NAME}-control-plane")
kubectl config set-cluster "kind-${CLUSTER_NAME}" --server="https://${container_ip}:6443" --kubeconfig="${KUBECONFIG}"
With this configuration I can do the curl requests, however when I try to access the cluster via kubectl, I get a Unable to connect to the server: x509
.
If I can get this to work, I could leave out the "-L" option of the SSH tunneling, however I would still have the issue with the remote (EC2 instance) communication with the local api-server "-R" option, due to having NAT, but could it just work by setting 0.0.0.0
when creating the local KinD cluster setting it in localhost:6443 and from the remote EC2 accessing it with api-server 127.0.0.1:5252
?
I now also suspect that I might have 6443 port closed in the new local machine I am testing. So hopefully tomorrow I can ask for them to check the port and test it directly.
By your last response I suppose that setting the service could create more issues than facilitating the problem right?
Thank you very much for your help.
I am trying to join two kind clusters using Karmada Push mode, a kind cluster in my local PC (which is in the private network of my organization), and a kind cluster in an AWS EC2 instance.
After I set up a bidirectional SSH tunnel, I try to join the local cluster to the remote cluster using Push mode (from the EC2), it says that it has been joined succesfully, but the Ready state is in False.
I am going to describe the steps followed in order to reproduce the issue (LOCAL is my local machine and REMOTE is the EC2 instance).
*** LOCAL:
=> OK!
*** REMOTE:
=> OK!
// I copy the kubeconfig of localcluster to the remote EC2 (.kube/config-local) // I copy the kubeconfig of remotecluster and karmada-apiserver to local machine // In order to be able to access the clusters in both directions, ssh tunnels are opened in the following way // In local config: cluster: certificate-authority-data:
server: http://127.0.0.1:8765 name: karmada-apiserver // In remote config: cluster: certificate-authority-data: server: http://127.0.0.1:5252 name: kind-localcluster
*** LOCAL:
*** REMOTE:
*** LOCAL:
=> OK!
*** REMOTE:
=> OK!
*** LOCAL:
=> OK!
*** REMOTE:
In conclusion, it says that the cluster is not reachable, but I can access both clusters from each sides.
Is it a bug, or how can I solve this issue?
Thank you very much.
Environment:
kubectl karmada version kubectl karmada version: version.Info{GitVersion:"v1.6.0", GitCommit:"6eb79b38949e480cf7a2e12cfa56fef47bda81ea", GitTreeState:"clean", BuildDate:"2023-06-02T08:04:58Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"linux/amd64"}
karmadactl version karmadactl version: version.Info{GitVersion:"v1.6.0", GitCommit:"6eb79b38949e480cf7a2e12cfa56fef47bda81ea", GitTreeState:"clean", BuildDate:"2023-05-31T09:55:29Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"linux/amd64"}
kind version // The same in local and remote kind v0.17.0 go1.19.2 linux/amd64
KIND CLUSTERS config file:
*kind-config.yaml:
$ kubectl version // (Local machine) WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0", GitCommit:"b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d", GitTreeState:"clean", BuildDate:"2022-12-08T19:58:30Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.7 Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.4", GitCommit:"f89670c3aa4059d6999cb42e23ccb4f0b9a03979", GitTreeState:"clean", BuildDate:"2023-05-17T00:01:22Z", GoVersion:"go1.19.8", Compiler:"gc", Platform:"linux/amd64"}
kubectl version // (Remote EC2) WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:40:17Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.7 Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.4", GitCommit:"872a965c6c6526caa949f0c6ac028ef7aff3fb78", GitTreeState:"clean", BuildDate:"2022-11-09T13:29:58Z", GoVersion:"go1.19.3", Compiler:"gc", Platform:"linux/amd64"}