search

shanejansen / touchstone

Touchstone is a testing framework for your services that focuses on component, end-to-end, and exploratory testing.
16 stars 2 forks source link

Service logs do not cover immediate startup failures #36

Closed ScottFreeCode closed 2 years ago

ScottFreeCode commented 2 years ago

Usually when a service fails to pass the healthcheck, --log-services provides useful info. However, sometimes the log file is empty and to see its output I have to catch the container with a Docker attach command in the fleeting window during which it is live. It seems as though Touchstone connects to the container after the initial docker run command to get the logs. I would expect it to direct the container output to a file as part of the initial container run/start command, guaranteeing all output is logged.

shanejansen commented 2 years ago

Log files are retrieved using the docker container logs command just before the container is stopped. Not sure why that wouldn't be capturing all logs. May have to do some investigation

ScottFreeCode commented 2 years ago

Possibly the container is already dead when Touchstone attempts the command; it really only happens when the app fails to start up in the first place (but if I can catch it while it's trying to start I do see that it's logging, it's just that it's already crashed by the time Touchstone tries to shut it down).

(On the other hand if it's stopped but the container should still have its logs retrievable, and Docker still isn't providing them, maybe it's a Docker bug!)

ScottFreeCode commented 2 years ago

It occurred to me rereading this that it's possible Touchstone may be starting the container with the autoremove flag, in which case removing that flag and additing explicit container cleanup after/during shutdown could resolve this.

(Speaking of which, does Touchstone clean up built services' intermediate images/layers? Obviously calling prune and potentially catching non-Touchstone stuff would be inadvisable; but as regards its own service images, the more Touchstone can clean up after itself the more disk space will be!)

shanejansen commented 2 years ago

Touchstone does run containers with the --rm flag, so you're right the container could be removed before Touchstone has a change to get logs if a health check fails.

During shutdown, Touchstone will clean up images that it produced (like supplying a Dockerfile in touchstone.yml)

ScottFreeCode commented 2 years ago

I have opened two PRs with different possible fixes, with different pros and cons. #42 and #43 EDIT: wow, they fix most of each others' cons if combined, and I didn't even think about that!