albertodonato
commented
2 days ago Hi thanks for reporting this. I'm actually working on adding support for structured logging in prometheus-aioexporter (and to query-exporter as a consequence, with the option to log in JSON format.
I have work in progress branches for both, hopefully I should be able to get something out soon.
Is your feature request related to a problem? Please describe. Running this application in a docker container on Google Kubernetes Engine results in all logs (healthcheck logs, etc) being ingested as ERROR severity, since python logs to stderr by default. This is misleading since they are not actual errors.
Describe the solution you'd like I think the standard way to solve this is to emit structured JSON logs with a "severity" field (rather than just logging to stdout). If it would be disruptive to change this for users, then it would be nice to at least get an off-by-default config flag to enable structured logs.
Describe alternatives you've considered We could just hardcode the logger to write to stdout, but that seems heavyhanded.
We could configure our GKE-managed fluentbit to scrape logs from this container differently than all the others, setting the severity using some regex, but that's very fragile and a bunch of work.
Very open to alternatives!