search

canonical / microk8s

MicroK8s is a small, fast, single-package Kubernetes for datacenters and the edge.
https://microk8s.io
Apache License 2.0
8.51k stars 772 forks source link

error: unable to forward port because pod is not running. Current status=Pending #1547

Closed geewynn closed 2 years ago

geewynn commented 4 years ago

Hello, I started an ubuntu 18.04 virtual machine on google cloud platform with required vCPU, memory and disk storage, Once the machine was up, I ssh into it and installed microk8s. Then I deployed kubeflow using the link below

https://v1-0-branch.kubeflow.org/docs/started/k8s/kfctl-istio-dex/#expose-kubeflow

Now I am trying to access kubeflow with this instruction kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80 But it returns this error error: unable to forward port because pod is not running. Current status=Pending

Tarball from microk8s inspect inspection-report-20200908_182514.tar.gz

microk8s v1.18 kfctl - https://github.com/kubeflow/kfctl/releases/download/v1.0.2/kfctl_v1.0.2-0-ga476281_linux.tar.gz manifest - https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_istio_dex.v1.0.2.yaml

knkski commented 4 years ago

It looks like the pod is still booting up:

istio-system      pod/istio-ingressgateway-c79f9f6f-qj7z6                            0/1     ContainerCreating   0          9m50s

Given that there's no restarts on that pod, I would assume that it's taking a long time to boot up, as opposed to running into any errors. Are you able to wait for the pod(s) to finish booting up with microk8s kubectl wait -n istio-system --for=condition=ready pod --all before attempting the port forward?

If the pod never finishes starting up, can you run microk8s kubectl describe -n istio-system on the ingress-gateway Pod and Deployment resources and post the output here?

geewynn commented 4 years ago

Running this code microk8s kubectl wait -n istio-system --for=condition=ready pod --all the output.

pod/authservice-0 condition met
pod/istio-citadel-7f66ddfcfb-fr5xq condition met
pod/istio-galley-7976dd55cd-7md2k condition met
pod/istio-nodeagent-dnmfp condition met
pod/istio-sidecar-injector-56b6997f7d-kprdj condition met
pod/prometheus-65fdcbc857-smr9p condition met
timed out waiting for the condition on pods/cluster-local-gateway-84bb595449-8sstc
timed out waiting for the condition on pods/istio-ingressgateway-c79f9f6f-qj7z6
timed out waiting for the condition on pods/istio-pilot-7bd96d69d9-wdpt8
timed out waiting for the condition on pods/istio-policy-66b5d9887c-j6pkf
timed out waiting for the condition on pods/istio-security-post-install-release-1.3-latest-daily-tt6b9
timed out waiting for the condition on pods/istio-telemetry-856f7bcff4-4262c

It timed out on the pods taking time to run.

Pod description

Name:           istio-ingressgateway-c79f9f6f-qj7z6
Namespace:      istio-system
Priority:       0
Node:           cubeflow/10.138.0.2
Start Time:     Tue, 08 Sep 2020 18:15:22 +0000
Labels:         app=istio-ingressgateway
                istio=ingressgateway
                pod-template-hash=c79f9f6f
Annotations:    sidecar.istio.io/inject: false
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  ReplicaSet/istio-ingressgateway-c79f9f6f
Containers:
  istio-proxy:
    Container ID:  
    Image:         gcr.io/istio-release/proxyv2:release-1.3-latest-daily
    Image ID:      
    Ports:         15020/TCP, 80/TCP, 443/TCP, 31400/TCP, 15029/TCP, 15030/TCP, 15031/TCP, 15032/TCP, 15443/TCP, 15090/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
      proxy
      router
      --domain
      $(POD_NAMESPACE).svc.cluster.local
      --log_output_level=default:info
      --drainDuration
      45s
      --parentShutdownDuration
      1m0s
      --connectTimeout
      10s
      --serviceCluster
      istio-ingressgateway
      --zipkinAddress
      zipkin:9411
      --proxyAdminPort
      15000
      --statusPort
      15020
      --controlPlaneAuthPolicy
      MUTUAL_TLS
      --discoveryAddress
      istio-pilot:15011
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:      100m
      memory:   128Mi
    Readiness:  http-get http://:15020/healthz/ready delay=1s timeout=1s period=2s #success=1 #failure=30
    Environment:
      NODE_NAME:                     (v1:spec.nodeName)
      POD_NAME:                     istio-ingressgateway-c79f9f6f-qj7z6 (v1:metadata.name)
      POD_NAMESPACE:                istio-system (v1:metadata.namespace)
      INSTANCE_IP:                   (v1:status.podIP)
      HOST_IP:                       (v1:status.hostIP)
      SERVICE_ACCOUNT:               (v1:spec.serviceAccountName)
      ISTIO_META_POD_NAME:          istio-ingressgateway-c79f9f6f-qj7z6 (v1:metadata.name)
ISTIO_META_POD_NAME:          istio-ingressgateway-c79f9f6f-qj7z6 (v1:metadata.name)
      ISTIO_META_CONFIG_NAMESPACE:  istio-system (v1:metadata.namespace)
      SDS_ENABLED:                  true
      ISTIO_META_WORKLOAD_NAME:     istio-ingressgateway
      ISTIO_META_OWNER:             kubernetes://api/apps/v1/namespaces/istio-system/deployments/istio-ingressgateway
      ISTIO_META_ROUTER_MODE:       sni-dnat
    Mounts:
      /etc/certs from istio-certs (ro)
      /etc/istio/ingressgateway-ca-certs from ingressgateway-ca-certs (ro)
      /etc/istio/ingressgateway-certs from ingressgateway-certs (ro)
      /var/run/sds from sdsudspath (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from istio-ingressgateway-service-account-token-wscf5 (ro)
      /var/run/secrets/tokens from istio-token (rw)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  sdsudspath:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/sds
    HostPathType:  
  istio-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  43200
  istio-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  43200
  istio-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio.istio-ingressgateway-service-account
    Optional:    true
  ingressgateway-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio-ingressgateway-certs
    Optional:    true
  ingressgateway-ca-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio-ingressgateway-ca-certs
    Optional:    true
  istio-ingressgateway-service-account-token-wscf5:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio-ingressgateway-service-account-token-wscf5
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age                    From               Message
  ----     ------       ----                   ----               -------
  Warning  FailedMount  48m (x7 over 109m)     kubelet, cubeflow  Unable to attach or mount volumes: unmounted volumes=[istio-token], unattached volumes=[ingressgateway-certs ingressgateway-ca-certs istio-ingressgateway-service-account-token-wscf5 sdsudspath istio-token istio-certs]: timed out waiting for the condition
  Warning  FailedMount  28m (x11 over 116m)    kubelet, cubeflow  Unable to attach or mount volumes: unmounted volumes=[istio-token], unattached volumes=[sdsudspath istio-token istio-certs ingressgateway-certs ingressgateway-ca-certs istio-ingressgateway-service-account-token-wscf5]: timed out waiting for the condition
  Warning  FailedMount  19m (x9 over 121m)     kubelet, cubeflow  Unable to attach or mount volumes: unmounted volumes=[istio-token], unattached volumes=[istio-ingressgateway-service-account-token-wscf5 sdsudspath istio-token istio-certs ingressgateway-certs ingressgateway-ca-certs]: timed out waiting for the condition
  Warning  FailedMount  14m (x7 over 123m)     kubelet, cubeflow  Unable to attach or mount volumes: unmounted volumes=[istio-token], unattached volumes=[istio-token istio-certs ingressgateway-certs ingressgateway-ca-certs istio-ingressgateway-service-account-token-wscf5 sdsudspath]: timed out waiting for the condition
  Warning  FailedMount  3m29s (x70 over 129m)  kubelet, cubeflow  MountVolume.SetUp failed for volume "istio-token" : failed to fetch token: the API server does not have TokenRequest endpoints enabled
ktsakalozos commented 4 years ago

Hi @geewynn, does this https://github.com/kubeflow/manifests/issues/959 look related?

davidspek commented 4 years ago

@ktsakalozos I was facing a somewhat related issue to the one you linked when trying to deploy kubeflow on Charmed Kubernetes https://github.com/juju-solutions/bundle-kubeflow/issues/240#issuecomment-692758081.

shahzaibekram commented 3 years ago

I have the same problem with my microk8s pods and I have done all solution but still pods not running

AMESPACE     NAME                                         READY   STATUS              RESTARTS   AGE
kube-system   metrics-server-8bbfb4bdb-q9ldh               0/1     ContainerCreating   0          29m
kube-system   kubernetes-dashboard-7ffd448895-qm44w        0/1     ContainerCreating   0          28m
kube-system   dashboard-metrics-scraper-6c4568dc68-qrcjv   0/1     ContainerCreating   0          28m
kube-system   coredns-86f78bb79c-v7tgt                     0/1     ContainerCreating   0          31m
kube-system   calico-kube-controllers-847c8c99d-g59cp      0/1     ContainerCreating   0          32m
kube-system   calico-node-xph9f                            0/1     CrashLoopBackOff    10         32m
davidspek commented 3 years ago

@shahzaibekram Did you use microk8s enable kubeflow by any chance? Because for issues related to that you should be over at https://github.com/juju-solutions/bundle-kubeflow/issues

shahzaibekram commented 3 years ago

My pod logs what can I do to solve this problem 

Name:           kubernetes-dashboard-7ffd448895-qm44w
Namespace:      kube-system
Priority:       0
Node:           shah/192.168.0.104
Start Time:     Thu, 28 Jan 2021 17:54:37 +0500
Labels:         k8s-app=kubernetes-dashboard
                pod-template-hash=7ffd448895
Annotations:    <none>
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  ReplicaSet/kubernetes-dashboard-7ffd448895
Containers:
  kubernetes-dashboard:
    Container ID:  
    Image:         kubernetesui/dashboard:v2.0.0
    Image ID:      
    Port:          8443/TCP
    Host Port:     0/TCP
    Args:
      --auto-generate-certificates
      --namespace=kube-system
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Liveness:       http-get https://:8443/ delay=30s timeout=30s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /certs from kubernetes-dashboard-certs (rw)
      /tmp from tmp-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kubernetes-dashboard-token-m4mk2 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kubernetes-dashboard-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kubernetes-dashboard-certs
    Optional:    false
  tmp-volume:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kubernetes-dashboard-token-m4mk2:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kubernetes-dashboard-token-m4mk2
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                  From               Message
  ----     ------                  ----                 ----               -------
  Normal   Scheduled               30m                  default-scheduler  Successfully assigned kube-system/kubernetes-dashboard-7ffd448895-qm44w to shah
  Warning  FailedCreatePodSandBox  30m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "66da87b9899130a5eddd9f751a394ac529148fd0de3b8f5d01cdcc30688ffdd9": error getting ClusterInformation: resource does not exist: ClusterInformation(default) with error: clusterinformations.crd.projectcalico.org "default" not found
  Warning  FailedCreatePodSandBox  30m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9cd3282f69c89187a5c1cde109459aab287a0982cb35c42ec85273a82d971cc3": error getting ClusterInformation: resource does not exist: ClusterInformation(default) with error: clusterinformations.crd.projectcalico.org "default" not found
  Warning  FailedCreatePodSandBox  30m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "bae06850c830fcb711fcdf312be2ba220ca8e5e384bce747d2173343ecc0bb32": error getting ClusterInformation: resource does not exist: ClusterInformation(default) with error: clusterinformations.crd.projectcalico.org "default" not found
  Warning  FailedCreatePodSandBox  30m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b93f01f89a3aaea5d30ea25ee82debfccc1bd3ad7f0857fd7399bcc5c074935f": error getting ClusterInformation: resource does not exist: ClusterInformation(default) with error: clusterinformations.crd.projectcalico.org "default" not found
  Warning  FailedCreatePodSandBox  29m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ac9f00ee49b38b6471ca48b8e3e6f27c5725faaacebaef508ed50e601f2db2d8": error getting ClusterInformation: resource does not exist: ClusterInformation(default) with error: clusterinformations.crd.projectcalico.org "default" not found
  Warning  FailedCreatePodSandBox  29m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6bdcaa77db2c43bc59cd2c1e818e681ecf1543c48fdff5784e88c092f647483e": error getting ClusterInformation: resource does not exist: ClusterInformation(default) with error: clusterinformations.crd.projectcalico.org "default" not found
  Warning  FailedCreatePodSandBox  29m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6190c3c539440ec1200204fb2f5c69e8ca2a4e52fe8456ab3ddc2b6bc693c77f": error getting ClusterInformation: resource does not exist: ClusterInformation(default) with error: clusterinformations.crd.projectcalico.org "default" not found
  Warning  FailedCreatePodSandBox  29m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2c5b173879831db79ccf573506a28c57acd47514012a7e743240fee1c8b1a12f": error getting ClusterInformation: resource does not exist: ClusterInformation(default) with error: clusterinformations.crd.projectcalico.org "default" not found
  Warning  FailedCreatePodSandBox  29m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7c8ec308c522f2688445509ad8b157b67953a224e40d5ee3dfe71ced8349acdd": error getting ClusterInformation: resource does not exist: ClusterInformation(default) with error: clusterinformations.crd.projectcalico.org "default" not found
  Warning  FailedCreatePodSandBox  49s (x126 over 28m)  kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "31868e97def1e40694d30f8f481cbfe99abf2c0b0fa5032ab5369e33ccf81b4c": error getting ClusterInformation: resource does not exist: ClusterInformation(default) with error: clusterinformations.crd.projectcalico.org "default" not found
shahzaibekram commented 3 years ago

@shahzaibekram Did you use microk8s enable kubeflow by any chance? Because for issues related to that you should be over at https://github.com/juju-solutions/bundle-kubeflow/issues

Yes I used but didn't work

knkski commented 3 years ago

@shahzaibekram: can you open a new issue and ping @ktsakalozos on it? The kube-system calico-node-xph9f 0/1 CrashLoopBackOff 10 32m line makes it look like a networking issue.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

dayo777 commented 10 months ago

(Ubuntu 22.04.3 LTS) I also ran into this issue, and in my case, it seemed Jenkins was running locally on the same port. I had to stop Jenkins to work around it. Another workaround was using a different port number.