Every product should log diagnostic information about its environment on startup

stackabletech / issues

This repository is only for issues that concern multiple repositories or don't fit into any specific repository

2 stars 0 forks source link

Every product should log diagnostic information about its environment on startup #527

Open lfrancke opened 8 months ago

lfrancke commented 8 months ago

As a user and admin of our products I'd like to get information about the environment of a container to help me in debugging.

In particular I'd like to have a log of my own IP address(es), my hostname, forward and reverse lookup done using system tools as well as python and Java (to catch any errors), the current time and any other non-sensitive information.

soenkeliebau commented 5 months ago

Talking to a customer yesterday, we should probably log this information regularly instead of only at startup. Some containers will run a fair amount of time and they throw away logs from their log management system after 7 days, so startup messages may not be available anymore at the time of a support ticket. Maybe daily or even hourly on the full hour makes sense to make it easy to include in log dumps and findable for the person looking at logs.

lfrancke commented 3 months ago

If possible it'd be great to also output information about any truststores and keystores in use (obviously we need to make sure no sensitive material is logged!)

nightkr commented 5 days ago

After investigating a bit (see also: https://github.com/stackabletech/decisions/issues/5#issuecomment-2413908310), I'd propose the folllowing:

Add containerdebug to the stackable-base Docker image
Add containerdebug --loop & to the into sequence of the main containers (where --loop is a hypothetical argument for https://github.com/stackabletech/containerdebug/issues/7)
- This way, containerdebug runs in exactly the same environment as the main application (with the same mounts, memory allocations, etc etc etc)
- This also means that containerdebug dies when the app dies (container is terminated) but not the other way around (obviously we'd prefer if containerdebug kept running forever, but running the main app is still by far more important)
containerdebug should output to stdout (for kubectl logs, merged with the app logs) and vector (separate, to help vector parse it and redirect to whatever backend storage)

lfrancke commented 5 days ago

That makes sense to me. I don't think we need a new decision for this but I'd highlight this on Slack so people are aware.

The only question I have is whether the stdout should be a different format than the vector output?

nightkr commented 4 days ago

I would say use a human-oriented "loggy" format for both. Maybe a machine-oriented JSON output to a file somewhere.