Liveness probe fails

[mscbpi]

I tried to deploy PSO-Explorer without success.

$ kubectl get po -n pso-explorer
NAME                            READY   STATUS             RESTARTS   AGE
pso-explorer-649f8b9767-pffpq   1/2     CrashLoopBackOff   6          9m4s

$kubectl logs -n pso-explorer pso-explorer-649f8b9767-pffpq pso-explorer
Starting nginx: nginx.
[20-Jul-2020 07:32:56] NOTICE: [pool www] 'user' directive is ignored when FPM is not running as root
[20-Jul-2020 07:32:56] NOTICE: [pool www] 'user' directive is ignored when FPM is not running as root
[20-Jul-2020 07:32:56] NOTICE: [pool www] 'group' directive is ignored when FPM is not running as root
[20-Jul-2020 07:32:56] NOTICE: [pool www] 'group' directive is ignored when FPM is not running as root
[20-Jul-2020 07:32:56] NOTICE: fpm is running, pid 24
[20-Jul-2020 07:32:56] NOTICE: ready to handle connections

$ kubectl describe po -n pso-explorer  pso-explorer-649f8b9767-pffpq
Name:         pso-explorer-649f8b9767-pffpq
Namespace:    pso-explorer
Priority:     0
Node:         mdc-k8s-worker5/10.129.4.17
Start Time:   Mon, 20 Jul 2020 09:27:32 +0200
Labels:       app.kubernetes.io/instance=pso-explorer
              app.kubernetes.io/name=pso-explorer
              pod-template-hash=649f8b9767
Annotations:  cni.projectcalico.org/podIP: 10.42.8.38/32
              cni.projectcalico.org/podIPs: 10.42.8.38/32
Status:       Running
IP:           10.42.8.38
IPs:
  IP:           10.42.8.38
Controlled By:  ReplicaSet/pso-explorer-649f8b9767
Containers:
  pso-explorer:
    Container ID:   docker://39df3e46771ee34cd4ccd0ca9b3ee3229b94dee2ed22c891531921a280abfc7b
    Image:          quay.io/purestorage/pso-explorer:v0.5.3-devel
    Image ID:       docker-pullable://quay.io/purestorage/pso-explorer@sha256:3e21dc4c436b60871edf22c08f06e5f4f8194ca333c241dab4d9a686935ec4c0
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Mon, 20 Jul 2020 09:35:29 +0200
      Finished:     Mon, 20 Jul 2020 09:36:23 +0200
    Ready:          False
    Restart Count:  6
    Liveness:       http-get http://:http/ delay=0s timeout=10s period=10s #success=1 #failure=3
    Readiness:      http-get http://:http/ delay=0s timeout=10s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from pso-explorer-token-bt59n (ro)
  redis:
    Container ID:   docker://40bb06014b1be15a200e76871e71edea70195c9da17ff786cfc0e21310c068c7
    Image:          redis:alpine
    Image ID:       docker-pullable://redis@sha256:b7561e4e994ab52cb2062b9c3e943ab2af3287caf9d9aeb6719acc77013769a5
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Mon, 20 Jul 2020 09:27:56 +0200
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from pso-explorer-token-bt59n (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  pso-explorer-token-bt59n:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  pso-explorer-token-bt59n
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                     From                      Message
  ----     ------     ----                    ----                      -------
  Normal   Scheduled  <unknown>               default-scheduler         Successfully assigned pso-explorer/pso-explorer-649f8b9767-pffpq to mdc-k8s-worker5
  Normal   Pulling    9m23s                   kubelet, mdc-k8s-worker5  Pulling image "redis:alpine"
  Normal   Created    9m19s                   kubelet, mdc-k8s-worker5  Created container redis
  Normal   Pulled     9m19s                   kubelet, mdc-k8s-worker5  Successfully pulled image "redis:alpine"
  Normal   Started    9m18s                   kubelet, mdc-k8s-worker5  Started container redis
  Normal   Killing    8m52s                   kubelet, mdc-k8s-worker5  Container pso-explorer failed liveness probe, will be restarted
  Normal   Pulling    8m22s (x2 over 9m42s)   kubelet, mdc-k8s-worker5  Pulling image "quay.io/purestorage/pso-explorer:v0.5.3-devel"
  Normal   Created    8m20s (x2 over 9m23s)   kubelet, mdc-k8s-worker5  Created container pso-explorer
  Normal   Started    8m20s (x2 over 9m23s)   kubelet, mdc-k8s-worker5  Started container pso-explorer
  Normal   Pulled     8m20s (x2 over 9m24s)   kubelet, mdc-k8s-worker5  Successfully pulled image "quay.io/purestorage/pso-explorer:v0.5.3-devel"
  Warning  Unhealthy  8m2s (x5 over 9m12s)    kubelet, mdc-k8s-worker5  Liveness probe failed: Get http://10.42.8.38:80/: dial tcp 10.42.8.38:80: connect: connection refused
  Warning  Unhealthy  4m41s (x28 over 9m11s)  kubelet, mdc-k8s-worker5  Readiness probe failed: Get http://10.42.8.38:80/: dial tcp 10.42.8.38:80: connect: connection refused

[mscbpi]

For some reason http container port was declared as 80 while according to deployment.yaml it should have been 8080. manually setting it makes the pso-explorer work.

[rdeenik]

Hi @mscbpi, It seems you are trying to run the 0.5.3-devel build, which might indeed still have some issues. Have you also tried using the 0.5.2 release?

[mscbpi]

Indeed, I took default values.yaml (just changing svc to clusterIP). Noticing the -devel I tried previous version, works out of the box with 0.5.2 :)

[sdodsley]

Not an issue in the released version.