Stunner service still in pending status

[krajcikondra]

Hi,

I have similar issue as my previous https://github.com/l7mp/stunner/issues/96

I installed and configured stunner but my stunner service still in pending status.

$ kubectl get pods -n stunner
NAME                       READY   STATUS    RESTARTS   AGE
stunner-7ff4875b47-l9jsp   0/2     Pending   0          6m22s

I am using DOKS (Digital ocean Kubernetes ).

There is some way how debug my stunner service?

[levaitamas]

Hi @krajcikondra,

If I get this correctly, you applied the changes mentioned in #96 and now stunner does not start on DOKS. Just to double-check: this is a clean deployment (previous stunner installations are deleted)?

Could you please show the output of these commands (without any sensitive information like public IP, etc.)?

• kubectl describe pod -n stunner stunner-7ff4875b47-l9jsp
• kubectl logs -n stunner stunner-7ff4875b47-l9jsp

[krajcikondra]

No, I applied changes mentioned in #96 but stunner service has never worked.

$ kubectl describe pod -n stunner stunner-7ff4875b47-l9jsp
Name:           stunner-7ff4875b47-l9jsp
Namespace:      stunner
Priority:       0
Node:           <none>
Labels:         app=stunner
                app.kubernetes.io/instance=stunner
                app.kubernetes.io/managed-by=Helm
                app.kubernetes.io/name=stunner
                pod-template-hash=7ff4875b47
Annotations:    <none>
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  ReplicaSet/stunner-7ff4875b47
Containers:
  stunnerd:
    Image:      l7mp/stunnerd:0.15.0
    Port:       8080/TCP
    Host Port:  0/TCP
    Command:
      stunnerd
    Args:
      -w
      -c
      /etc/stunnerd/stunnerd.conf
      --udp-thread-num=16
    Limits:
      cpu:     2
      memory:  512Mi
    Requests:
      cpu:      500m
      memory:   128Mi
    Liveness:   http-get http://:8086/live delay=0s timeout=1s period=5s #success=1 #failure=3
    Readiness:  http-get http://:8086/ready delay=0s timeout=1s period=5s #success=1 #failure=3
    Environment:
      STUNNER_ADDR:   (v1:status.podIP)
    Mounts:
      /etc/stunnerd from stunnerd-config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tns6t (ro)
  config-watcher:
    Image:      kiwigrid/k8s-sidecar:latest
    Port:       <none>
    Host Port:  <none>
    Limits:
      cpu:     500m
      memory:  500Mi
    Requests:
      cpu:     100m
      memory:  128Mi
    Environment:
      LABEL:        stunner.l7mp.io/owned-by
      LABEL_VALUE:  stunner
      FOLDER:       /etc/stunnerd
      RESOURCE:     configmap
      NAMESPACE:    stunner
    Mounts:
      /etc/stunnerd from stunnerd-config-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tns6t (ro)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  stunnerd-config-volume:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-tns6t:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason             Age                     From                Message
  ----     ------             ----                    ----                -------
  Warning  FailedScheduling   4m25s (x202 over 104m)  default-scheduler   0/2 nodes are available: 2 Insufficient cpu. preemption: 0/2 nodes are available: 2 No preemption victims found for incoming pod.
  Normal   NotTriggerScaleUp  33m (x50 over 104m)     cluster-autoscaler  pod didn't trigger scale-up:

$ kubectl logs -n stunner stunner-7ff4875b47-l9jsp
error: a container name must be specified for pod stunner-7ff4875b47-l9jsp, choose one of: [stunnerd config-watcher]

Thanks, now I see there is insufficient cpu. I try increase cpu resources

[krajcikondra]

I bought new node (2cpu, 4GB RAM) only for stunner. But still same issue.

$ kubectl describe pod -n stunner stunner-7ff4875b47-t5w4p
Name:           stunner-7ff4875b47-t5w4p
Namespace:      stunner
Priority:       0
Node:           <none>
Labels:         app=stunner
                app.kubernetes.io/instance=stunner
                app.kubernetes.io/managed-by=Helm
                app.kubernetes.io/name=stunner
                pod-template-hash=7ff4875b47
Annotations:    <none>
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  ReplicaSet/stunner-7ff4875b47
Containers:
  stunnerd:
    Image:      l7mp/stunnerd:0.15.0
    Port:       8080/TCP
    Host Port:  0/TCP
    Command:
      stunnerd
    Args:
      -w
      -c
      /etc/stunnerd/stunnerd.conf
      --udp-thread-num=16
    Limits:
      cpu:     2
      memory:  512Mi
    Requests:
      cpu:      500m
      memory:   128Mi
    Liveness:   http-get http://:8086/live delay=0s timeout=1s period=5s #success=1 #failure=3
    Readiness:  http-get http://:8086/ready delay=0s timeout=1s period=5s #success=1 #failure=3
    Environment:
      STUNNER_ADDR:   (v1:status.podIP)
    Mounts:
      /etc/stunnerd from stunnerd-config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-frnkg (ro)
  config-watcher:
    Image:      kiwigrid/k8s-sidecar:latest
    Port:       <none>
    Host Port:  <none>
    Limits:
      cpu:     500m
      memory:  500Mi
    Requests:
      cpu:     100m
      memory:  128Mi
    Environment:
      LABEL:        stunner.l7mp.io/owned-by
      LABEL_VALUE:  stunner
      FOLDER:       /etc/stunnerd
      RESOURCE:     configmap
      NAMESPACE:    stunner
    Mounts:
      /etc/stunnerd from stunnerd-config-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-frnkg (ro)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  stunnerd-config-volume:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-frnkg:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  3m3s   default-scheduler  0/3 nodes are available: 1 node(s) had untolerated taint {node.cloudprovider.kubernetes.io/uninitialized: true}, 2 Insufficient cpu. preemption: 0/3 nodes are available: 1 Preemption is not helpful for scheduling, 2 No preemption victims found for incoming pod.
  Warning  FailedScheduling  2m44s  default-scheduler  0/3 nodes are available: 1 node(s) had untolerated taint {node.cloudprovider.kubernetes.io/uninitialized: true}, 2 Insufficient cpu. preemption: 0/3 nodes are available: 1 Preemption is not helpful for scheduling, 2 No preemption victims found for incoming pod.

Can somebody help me please?

[levaitamas]

Hi @krajcikondra,

It is interesting why k8s cannot schedule the stunner pod to any of the available nodes. I made a quick test in minikube (2 CPU, 4GB RAM) and had no issues due to limited CPU resources. Couldn't it be that your new node is the tainted one and that prohibits scheduling the stunner pod to it?

Anyway, I'm closing this issue for now since this problem is not really a STUNner issue. Feel free to drop by at our Discord if you have any further questions, we're happy to help. I'd also like to take the opportunity to mention that we are in the WebRTC+Kubernetes consultancy business so if you want to throw some $$ to your DOKS setup, feel free to contact us.

[krajcikondra]

Problem solved by buy another 2CPU 4GB node. I was on limits with only one 2CPU 4GB node.