Pods no longer showing termination messages

overmindtech / aws-source

Overmind source for AWS resources

Other

5 stars 4 forks source link

Pods no longer showing termination messages #596

Closed dylanratcliffe closed 1 week ago

dylanratcliffe commented 3 weeks ago

I'm not seeing termination error messages in AWS sources. We need to fix this as the user needs to be able to see the message for debugging. Errors should be reported to /dev/termination_log which is then picked up by k8s

tphoney commented 2 weeks ago

@dylanratcliffe we use the same termination hook logic in revlink and k8s-source. have you noticed it happening with them as well

dylanratcliffe commented 2 weeks ago

I haven't checked revlink. We should use that logic for everything that runs as a pod realistically

tphoney commented 2 weeks ago

i confirmed the code is copy-pasta'ed so its either broken everywhere, or how they are launched, message read is broken.

tphoney commented 1 week ago

Other sources dockerfiles that use nobody : /workspace/cli/sources/k8s-source/build/package/Dockerfile /workspace/stdlib-source/build/package/Dockerfile

other services: /workspace/api-server/build/package/Dockerfile /workspace/gateway/build/package/Dockerfile /workspace/revlink/build/package/Dockerfile /workspace/srcman/build/package/Dockerfile

cli sources: /workspace/cli/sources/aws-source/build/package /workspace/cli/sources/k8s-source/build/package

We should also update the terraform in deploy to reflect the new termination-log location /workspace/deploy/modules/ovm-services/gateway.tf

tphoney commented 1 week ago

https://github.com/overmindtech/api-server/blob/1f0f906ef4ab168868e11a7a116e719c150a5783/server/source.go#L188-L196 is where we set the error message, but due to there being no log, and the field is omit empty. This never gets sent to the ui

then in k8

tphoney commented 1 week ago

deployment didnt work as expected. next steps. due to changes in not wanting to fatal in src's. this is for instrumentation purposes. we no longer hit the fatal / hook in root.go or we write an empty string to the log. we should

check if we actually need to change the output location of /dev/termination-log
keep the fallback to termination logs
for known errors we should write to the termination log, eg failing to connect to aws with credentials with a good error message. this will be displayed to the end user

tphoney commented 1 week ago