search

devfile / devworkspace-operator

Apache License 2.0
67 stars 55 forks source link

Output stdout & stderr of postStart events to Kubernetes event #1324

Closed AObuchow closed 3 weeks ago

AObuchow commented 1 month ago

Description

https://github.com/devfile/devworkspace-operator/pull/1081 added support for multiple postStart events. Their stdout and stderr are grouped together and redirected to /tmp/poststart-stdout.txt & /tmp/poststart-stderr.txt respectively.

Additionally, if multiple postStart events are used, the postStart hook will succeed as long as the last postStart event succeeds.

However, as a side effect, if the last (or only) postStart event fails, the stdout and stderr of the postStart event will not be propagated to the Kubernetes event on the cluster. Since a postStart hook failure will cause the pod to terminate, it's impossible to SSH/exec into the workspace container and read the output of /tmp/poststart-stdout.txt & /tmp/poststart-stderr.txt. This makes debugging failing postStart events difficult.

It'd be nice to somehow propagated the stdout & stderr of postStart events to the Kubernetes event, while also logging them into /tmp/poststart-stdout.txt & /tmp/poststart-stderr.txt .

Additional context

Downstream issue

AObuchow commented 3 weeks ago

It seems that an update that appeared in Kubernetes 1.26 now prevents the output of Kubernetes Lifecycle hooks from being displayed in the FailedPostStartHook event. This was done for security reasons to prevent any sensitive information from the postStart event from leaking.

Since this is outside of the control of DWO, I'm unfortunately going to close this issue.

AObuchow commented 2 weeks ago

Removing the v0.31.x milestone as this was closed as a WON'T FIX.