search

Azure / iot-edge-virtual-kubelet-provider

Kubernetes virtual kubelet provider for managing Azure IoT Edge deployments
MIT License
94 stars 25 forks source link

ProviderFailed error with virtual-kubelet on deploying sample-deployment.yaml #24

Open sibeshkar opened 5 years ago

sibeshkar commented 5 years ago

On deploying sample-deployment.yaml, the pod never starts, giving the ProviderFailed error. Have attached the output of kubectl describe pod here. Any help appreciated.

Name:           tempsensor-75869984f-wsp7s
Namespace:      default
Node:           iot-edge-connector-hub0/
Labels:         app=tempSensor
                pod-template-hash=314255409
Annotations:    isEdgeDeployment=true
                kubernetes.io/limit-ranger=LimitRanger plugin set: cpu request for container tempsensor
                loggingOptions=
                priority=15
                targetCondition=tags.type='bulb'
Status:         Pending
Reason:         ProviderFailed
Message:        500 Internal Server Error
IP:
Controlled By:  ReplicaSet/tempsensor-75869984f
Containers:
  tempsensor:
    Image:      mcr.microsoft.com/azureiotedge-simulated-temperature-sensor:1.0
    Port:       <none>
    Host Port:  <none>
    Requests:
      cpu:        100m
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-pgb24 (ro)
Conditions:
  Type           Status
  PodScheduled   True
Volumes:
  default-token-pgb24:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-pgb24
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  type=virtual-kubelet
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
                 virtual-kubelet.io/provider=iotedge:NoSchedule
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  7m    default-scheduler  Successfully assigned tempsensor-75869984f-wsp7s to iot-edge-connector-hub0
neumanndaniel commented 5 years ago

Have you modified the sample-deployment.yaml?

For me it's working out of the box without any issues.

Provider failed mostly indicates an improperly formatted yaml template file.

neumanndaniel commented 5 years ago

@sibeshkar Depending on how the configMap configuration looks like in your yaml template file, it might be that PR #25 fix the issue you are having.

vijaykatam commented 5 years ago

I saw the same issue. I tried with minikube, docker-desktop and finally AKS itself. In minikube and docker for desktop the kube-proxy pod running on virtual kubelet fails with ProviderError. The log message is Error from server: no preferred addresses found; known addresses: []

➜  k get nodes -owide                
NAME                       STATUS   ROLES   AGE     VERSION                             INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
aks-agentpool-27304320-0   Ready    agent   20m     v1.13.10                            10.240.0.4    <none>        Ubuntu 16.04.6 LTS   4.15.0-1052-azure   docker://3.0.6
iot-edge-connector-hub0    Ready    agent   3m37s   v1.13.1-vk-v0.9.0-1-g7b92d1ee-dev   <none>        <none>        <unknown>            <unknown>           <unknown>
➜ k get pods                        
NAME                                       READY   STATUS    RESTARTS   AGE
hub0-iot-edge-connector-6f79d6bdc7-lx64c   2/2     Running   0          4m20s
tempsensor-75564b4567-4fnr2                0/1     Running   0          2m16s
➜  k logs tempsensor-75564b4567-4fnr2
Error from server: no preferred addresses found; known addresses: []
neumanndaniel commented 5 years ago

@vijaykatam You are having a different issue, than original described herein.

Your tempsensor pod is running. The issue mentioned originally is where the tempsensor pod is stucked in the status pending and you get a ProviderFailed error with 500 Internal Server. That is most likely due to an improperly formatted yaml template file.

Your use case is not supported by the IoT Edge VK provider. The tempsensor pod is not really running on the Kubernetes cluster itself instead it is running on the IoT Edge device.

hacker-h commented 5 years ago

I'm having the exact same issue after following the docs. Additionally the iot edge connector installed with the helm chart from latest master turns unhealthy.

$ kubectl describe po

Name:           hub0-iot-edge-connector-775d5f5c79-rf4cr
Namespace:      default
Priority:       0
Node:           aks-agentpool-31716436-0/10.240.0.4
Start Time:     Tue, 15 Oct 2019 13:47:10 +0000
Labels:         app=iot-edge-connector
                pod-template-hash=775d5f5c79
                release=hub0
Annotations:    <none>
Status:         Running
IP:             10.244.0.14
IPs:            <none>
Controlled By:  ReplicaSet/hub0-iot-edge-connector-775d5f5c79
Containers:
  edgeprovider:
    Container ID:   docker://b443ec50c5fa1758253d0c5326d8c0e90d5c096ed46f36f3867ef7f26857da8b
    Image:          microsoft/iot-edge-vk-provider:latest
    Image ID:       docker-pullable://microsoft/iot-edge-vk-provider@sha256:1adf2a917819265d97e57023dddf78a7096d3791c7dcb67abb3955e16dafcdd5
    Port:           5000/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Tue, 15 Oct 2019 13:47:13 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  250Mi
    Requests:
      cpu:      10m
      memory:   120Mi
    Liveness:   http-get http://:5000/ delay=5s timeout=10s period=10s #success=1 #failure=3
    Readiness:  http-get http://:5000/ delay=5s timeout=10s period=10s #success=1 #failure=3
    Environment:
      HUB_CONNECTION_STRING:  <set to the key 'hub0-cs' in secret 'my-secrets'>  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from hub0-iot-edge-connector-service-account-token-wgg82 (ro)
  virtualkubelet:
    Container ID:  docker://b154fd2dff142df58834f121e0740a1e430531d00e443c62fc7fe467e4e6ba75
    Image:         microsoft/virtual-kubelet:latest
    Image ID:      docker-pullable://microsoft/virtual-kubelet@sha256:efc397d741d7e590c892c0ea5dccc9a800656c3adb95da4dae25c1cdd5eb6d9f
    Port:          <none>
    Host Port:     <none>
    Command:
      virtual-kubelet
    Args:
      --provider
      web
      --namespace

      --nodename
      iot-edge-connector-hub0
      --os
      Linux
    State:          Running
      Started:      Tue, 15 Oct 2019 13:47:14 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  100Mi
    Requests:
      cpu:     10m
      memory:  25Mi
    Environment:
      KUBELET_PORT:           10250
      VKUBELET_TAINT_KEY:     virtual-kubelet.io/provider
      VKUBELET_TAINT_VALUE:   iotedge
      VKUBELET_TAINT_EFFECT:  NoSchedule
      WEB_ENDPOINT_URL:       http://localhost:5000
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from hub0-iot-edge-connector-service-account-token-wgg82 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  hub0-iot-edge-connector-service-account-token-wgg82:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  hub0-iot-edge-connector-service-account-token-wgg82
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  beta.kubernetes.io/os=linux
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age   From                               Message
  ----     ------     ----  ----                               -------
  Normal   Scheduled  13m   default-scheduler                  Successfully assigned default/hub0-iot-edge-connector-775d5f5c79-rf4cr to aks-agentpool-31716436-0
  Normal   Pulling    13m   kubelet, aks-agentpool-31716436-0  pulling image "microsoft/iot-edge-vk-provider:latest"
  Normal   Pulled     13m   kubelet, aks-agentpool-31716436-0  Successfully pulled image "microsoft/iot-edge-vk-provider:latest"
  Normal   Created    13m   kubelet, aks-agentpool-31716436-0  Created container
  Normal   Started    13m   kubelet, aks-agentpool-31716436-0  Started container
  Normal   Pulling    13m   kubelet, aks-agentpool-31716436-0  pulling image "microsoft/virtual-kubelet:latest"
  Normal   Pulled     13m   kubelet, aks-agentpool-31716436-0  Successfully pulled image "microsoft/virtual-kubelet:latest"
  Normal   Created    13m   kubelet, aks-agentpool-31716436-0  Created container
  Normal   Started    13m   kubelet, aks-agentpool-31716436-0  Started container
  Warning  Unhealthy  13m   kubelet, aks-agentpool-31716436-0  Liveness probe failed: Get http://10.244.0.14:5000/: dial tcp 10.244.0.14:5000: connect: connection refused
  Warning  Unhealthy  13m   kubelet, aks-agentpool-31716436-0  Readiness probe failed: Get http://10.244.0.14:5000/: dial tcp 10.244.0.14:5000: connect: connection refused

Name:           tempsensor-75564b4567-qw8rd
Namespace:      default
Priority:       0
Node:           iot-edge-connector-hub0/
Labels:         app=tempSensor
                pod-template-hash=75564b4567
Annotations:    isEdgeDeployment: true
                loggingOptions:
                priority: 15
                targetCondition: tags.building='b43'
Status:         Pending
Reason:         ProviderFailed
Message:        500 Internal Server Error
IP:
IPs:            <none>
Controlled By:  ReplicaSet/tempsensor-75564b4567
Containers:
  tempsensor:
    Image:        mcr.microsoft.com/azureiotedge-simulated-temperature-sensor:1.0
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-5xrd4 (ro)
Conditions:
  Type           Status
  PodScheduled   True
Volumes:
  default-token-5xrd4:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-5xrd4
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  type=virtual-kubelet
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
                 virtual-kubelet.io/provider=iotedge:NoSchedule
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  8m41s  default-scheduler  Successfully assigned default/tempsensor-75564b4567-qw8rd to iot-edge-connector-hub0
mariojacobo commented 4 years ago

same issue here, no changes done to the sample-deployment.yaml

$ kubectl logs tempsensor-5f5979868f-c29nv
Error from server: no preferred addresses found; known addresses: []