Sometimes PostStartHookError message is "" even though the event has a error log

[roffe]

I'm using a lot of (container lifecycle hooks)[https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks] and these will output errors if something goes wrong, sometimes the slackbot does not print the message

for example here it printed the bash error in full because the file isn't chmod +x:

Container entered status *PostStartHookError*
```Exec lifecycle hook ([/usr/bin/dumb-init -- /bin/bash -c sleep 10 && /etc/hooks.d/ValidateService.sh]) for Container "k8s-app" in Pod "k8s-app-6d9858ddb7-t2jqp_default(8e650045-641b-11e8-a16d-021a35178fd6)" failed - error: command '/usr/bin/dumb-init -- /bin/bash -c sleep 10 && /etc/hooks.d/ValidateService.sh' exited with 126: /bin/bash: /etc/hooks.d/ValidateService.sh: Permission denied
, message: "/bin/bash: /etc/hooks.d/ValidateService.sh: Permission denied\n"```
default/k8s-app-6d9858ddb7-t2jqp/k8s-app
Container entered status *PostStartHookError*
```Exec lifecycle hook ([/usr/bin/dumb-init -- /bin/bash -c sleep 10 && /etc/hooks.d/ValidateService.sh]) for Container "k8s-app" in Pod "k8s-app-6d9858ddb7-t2jqp_default(8e650045-641b-11e8-a16d-021a35178fd6)" failed - error: command '/usr/bin/dumb-init -- /bin/bash -c sleep 10 && /etc/hooks.d/ValidateService.sh' exited with 126: /bin/bash: /etc/hooks.d/ValidateService.sh: Permission denied
, message: "/bin/bash: /etc/hooks.d/ValidateService.sh: Permission denied\n"```

while when the script was executed and outputted a error slack message is ""

Container entered status *PostStartHookError*
```Exec lifecycle hook ([/usr/bin/dumb-init -- /bin/bash -c sleep 10 && bash /etc/hooks.d/ValidateService.sh]) for Container "k8s-app" in Pod "k8s-app-66f89b584f-5kstg_default(be87b3f7-641b-11e8-a16d-021a35178fd6)" failed - error: command '/usr/bin/dumb-init -- /bin/bash -c sleep 10 && bash /etc/hooks.d/ValidateService.sh' exited with 137: , message: ""```
default/k8s-app-66f89b584f-5kstg/k8s-app
Container entered status *PostStartHookError*
```Exec lifecycle hook ([/usr/bin/dumb-init -- /bin/bash -c sleep 10 && bash /etc/hooks.d/ValidateService.sh]) for Container "k8s-app" in Pod "k8s-app-66f89b584f-5kstg_default(be87b3f7-641b-11e8-a16d-021a35178fd6)" failed - error: command '/usr/bin/dumb-init -- /bin/bash -c sleep 10 && bash /etc/hooks.d/ValidateService.sh' exited with 137: , message: ""```

the event looks perfectly fine in k8s api when i get it with kubectl get events

49m         49m          1         k8s-app-66f89b584f-jhppk.153376985d155a0b   Pod          spec.containers{k8s-app}   Warning   FailedPostStartHook     kubelet, ip-10-0-116-153.eu-west-1.compute.internal   Exec lifecycle hook ([/usr/bin/dumb-init -- /bin/bash -c sleep 10 && bash /etc/hooks.d/ValidateService.sh]) for Container "k8s-app" in Pod "k8s-app-66f89b584f-jhppk_default(58e883ed-6420-11e8-a16d-021a35178fd6)" failed - error: command '/usr/bin/dumb-init -- /bin/bash -c sleep 10 && bash /etc/hooks.d/ValidateService.sh' exited with 1: , message: "{\n\"cmdline\": [\"/opt/k8s-app/k8s-app\"nTest case FAILED!\nExpected:\n{\n  \"labels\": [\n    {\n      \"foo\": 7.9950147\n    }\n  ],\n  \"tokens\": null\n}\nReceived:\nFAIL ME HARD\n"

[roffe]

looks like this is not activly maintained, will look for other solutions

[whs]

Sorry I missed your issue.

kube-slack does not read Kubernetes events. It read Kubernetes pod metadata (what you get in kubectl describe pod, or more precisely kubectl get pod -o yaml) and report the exact message Kubernetes reported as the reason.

I think this is working as intended. If the container refused to start due to entrypoint spawning problem Docker would report an error during docker start and Kubernetes just report whatever it gets. If the container starts but crash, that is asynchronous and the output should goes into the logging system, which pod metadata is not one.