search

grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.87k stars 3.45k forks source link

Non-error logs are printed to stderr #5331

Open ku1ik opened 2 years ago

ku1ik commented 2 years ago

Describe the bug

As in title. This makes alerting configuration based on log severity problematic because everything printed by loki itself seems to be an error. Most logging aggregation tools and platforms treat stderr output as level=error.

For example:

2022-02-06 11:42:17 | level=info ts=2022-02-06T10:42:17.760239159Z caller=table.go:362 msg="finished uploading table index_19029

^^^ not an error but it's logged to stderr.

To Reproduce

Steps to reproduce the behavior:

  1. Start Loki (2.4.2)
  2. Wait until it prints its standard log messages (e.g. db, table related info)

Expected behavior Informational logs are printed to stdout, and only errors are printed to stderr.

Environment:

muffl0n commented 2 years ago

This seems to be defined here: https://github.com/grafana/loki/blob/46956d0bf6367aa320de22eb42345b3edb0166ac/pkg/util/log/log.go#L46-L49 I'm not sure how one would fix this. Just changing the code to log.NewLogfmtLogger(log.NewSyncWriter(os.Stdout)) is wrong either, cause everything above "WARN" should be printed to stderr furthermore.

It's similar for promtail, btw.

munsayac13 commented 2 years ago

Im having similar issue. We are using loki with fluentbit. Loki version 2.5.0 and Fluentbit 2.1.0 under kubernetes cluster 1.21