Connection Refused - Githubissues

nayuta-ai commented 1 year ago

Error of connection refused encountered. I want to fix the error.

$ kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-minikube" | jq '.'
{
  "kind": "PodMetrics",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {
    "name": "kube-scheduler-minikube",
    "namespace": "kube-system",
    "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-minikube",
    "creationTimestamp": "2022-10-06T08:20:00Z"
  },
  "timestamp": "2022-10-06T08:19:53Z",
  "window": "30s",
  "containers": [
    {
      "name": "kube-scheduler",
      "usage": {
        "cpu": "1983325n",
        "memory": "56940Ki"
      }
    }
  ]
}
$ curl http://localhost:8080/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-minikube
curl: (7) Failed to connect to localhost port 8080: 接続を拒否されました

The detail of "kube-scheduler-minikube" shows below.

$ kubectl describe pods kube-scheduler-minikube -n kube-system
Name:                 kube-scheduler-minikube
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Node:                 minikube/192.168.49.2
Start Time:           Fri, 23 Sep 2022 16:31:57 +0900
Labels:               component=kube-scheduler
                      tier=control-plane
Annotations:          kubernetes.io/config.hash: b564e2b429a630c74cbb01d3c4b7a228
                      kubernetes.io/config.mirror: b564e2b429a630c74cbb01d3c4b7a228
                      kubernetes.io/config.seen: 2022-09-23T07:31:57.302417269Z
                      kubernetes.io/config.source: file
Status:               Running
IP:                   192.168.49.2
IPs:
  IP:           192.168.49.2
Controlled By:  Node/minikube
Containers:
  kube-scheduler:
    Container ID:  docker://296c38d00078567e88cc4a95e2929093fda8688c9adbf0bf9029a831492514bb
    Image:         registry.k8s.io/kube-scheduler:v1.25.0
    Image ID:      docker-pullable://registry.k8s.io/kube-scheduler@sha256:9330c53feca7b51b25e427fa96afd5d1460b3233e9fa92e20c895c067da56ac1
    Port:          <none>
    Host Port:     <none>
    Command:
      kube-scheduler
      --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
      --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
      --bind-address=127.0.0.1
      --kubeconfig=/etc/kubernetes/scheduler.conf
      --leader-elect=false
    State:          Running
      Started:      Thu, 06 Oct 2022 16:38:14 +0900
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 06 Oct 2022 01:15:10 +0900
      Finished:     Thu, 06 Oct 2022 02:00:35 +0900
    Ready:          True
    Restart Count:  6
    Requests:
      cpu:        100m
    Liveness:     http-get https://127.0.0.1:10259/healthz delay=10s timeout=15s period=10s #success=1 #failure=8
    Startup:      http-get https://127.0.0.1:10259/healthz delay=10s timeout=15s period=10s #success=1 #failure=24
    Environment:  <none>
    Mounts:
      /etc/kubernetes/scheduler.conf from kubeconfig (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  kubeconfig:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/scheduler.conf
    HostPathType:  FileOrCreate
QoS Class:         Burstable
Node-Selectors:    <none>
Tolerations:       :NoExecute op=Exists
Events:
  Type    Reason          Age   From     Message
  ----    ------          ----  ----     -------
  Normal  SandboxChanged  16h   kubelet  Pod sandbox changed, it will be killed and re-created.
  Normal  Pulled          16h   kubelet  Container image "registry.k8s.io/kube-scheduler:v1.25.0" already present on machine
  Normal  Created         16h   kubelet  Created container kube-scheduler
  Normal  Started         16h   kubelet  Started container kube-scheduler
  Normal  SandboxChanged  45m   kubelet  Pod sandbox changed, it will be killed and re-created.
  Normal  Pulled          45m   kubelet  Container image "registry.k8s.io/kube-scheduler:v1.25.0" already present on machine
  Normal  Created         45m   kubelet  Created container kube-scheduler
  Normal  Started         45m   kubelet  Started container kube-scheduler

Also, the other cluster components show below.

$ kubectl top pods
NAME                  CPU(cores)   MEMORY(bytes)   
app-96895944b-bjmgn   1m           0Mi             
app-96895944b-frgvk   1m           0Mi             
app-96895944b-tg4pl   1m           0Mi
$ kubectl get po -n kube-system
NAME                                        READY   STATUS    RESTARTS       AGE
coredns-565d847f94-mllpr                    1/1     Running   6 (15h ago)    13d
etcd-minikube                               1/1     Running   6 (15h ago)    13d
kube-apiserver-minikube                     1/1     Running   6 (51m ago)    13d
kube-controller-manager-minikube            1/1     Running   6 (15h ago)    13d
kube-proxy-xzqb2                            1/1     Running   6 (15h ago)    13d
kube-scheduler-minikube                     1/1     Running   6 (15h ago)    13d
metrics-server-5fb54c8b89-k22rv             1/1     Running   9 (50m ago)    13d
storage-provisioner                         1/1     Running   11 (50m ago)   13d
vpa-admission-controller-64cb9cfb4b-9h5lv   1/1     Running   6 (15h ago)    13d
vpa-recommender-6fdd69df6c-mw8tm            1/1     Running   6 (15h ago)    13d
vpa-updater-8567bff646-vzr7c                1/1     Running   6 (15h ago)    13d

ryuji0123 commented 1 year ago

Hmm, let's check if we can access Kube-scheduler without any authentication first.

nayuta-ai commented 1 year ago

It can't access Kube-scheduler.

$ curl http://192.168.49.2
curl: (7) Failed to connect to 192.168.49.2 port 80: 接続を拒否されました
$ curl https://127.0.0.1:10259/healthz
curl: (7) Failed to connect to 127.0.0.1 port 10259: 接続を拒否されました

Also, I found not opening the external IP.

$ kubectl get services
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP          13d
my-service   NodePort    10.104.244.47   <none>        8080:30300/TCP   9d

But, the error of connection refused encountered.

$ minikube tunnel
Status: 
    machine: minikube
    pid: 173767
    route: 10.96.0.0/12 -> 192.168.49.2
    minikube: Running
    services: []
    errors: 
        minikube: no errors
        router: no errors
        loadbalancer emulator: no errors
$ curl http://192.168.49.2
curl: (7) Failed to connect to 192.168.49.2 port 80: 接続を拒否されました
$ curl https://127.0.0.1:10259/healthz
curl: (7) Failed to connect to 127.0.0.1 port 10259: 接続を拒否されました

nayuta-ai commented 1 year ago

It can't access Kube-scheduler.

$ curl http://192.168.49.2
curl: (7) Failed to connect to 192.168.49.2 port 80: 接続を拒否されました
$ curl https://127.0.0.1:10259/healthz
curl: (7) Failed to connect to 127.0.0.1 port 10259: 接続を拒否されました

Also, I found not opening the external IP.

$ kubectl get services
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP          13d
my-service   NodePort    10.104.244.47   <none>        8080:30300/TCP   9d

But, the error of connection refused encountered.

$ minikube tunnel
Status: 
    machine: minikube
    pid: 173767
    route: 10.96.0.0/12 -> 192.168.49.2
    minikube: Running
    services: []
    errors: 
        minikube: no errors
        router: no errors
        loadbalancer emulator: no errors
$ curl http://192.168.49.2
curl: (7) Failed to connect to 192.168.49.2 port 80: 接続を拒否されました
$ curl https://127.0.0.1:10259/healthz
curl: (7) Failed to connect to 127.0.0.1 port 10259: 接続を拒否されました

nayuta-ai / cloud_storage

Connection Refused #3