search

prometheus / prometheus

The Prometheus monitoring system and time series database.
https://prometheus.io/
Apache License 2.0
54.88k stars 9.04k forks source link

Patch log level config on runtime #10352

Open alileza opened 2 years ago

alileza commented 2 years ago

Proposal

Use case. Why is this important?

In some cases we would like to know what's going on with prometheus in production, we would like to see prometheus log change to debug. I thought it would be nice to have a way to change the log level in runtime without interrupting prometheus work (such as deployment).

So I'm thinking something like adding PATCH /config endpoint to prometheus could be nice, or maybe just specific only for the log level

roidelapluie commented 2 years ago

Yes, that's something I would like to add to Prometheus. it would involve moving the log level to file. this should be discussed in the dev summit because I don't know if everyone is happy with deprecating the command line flag.

krajorama commented 2 weeks ago

Hello from the bug scrub.

This was not yet discussed at the dev summit, we're adding it to the agend for the next in-person dev summit.

beorn7 commented 2 days ago

Update from the dev summit:

We finally discussed this. We agreed to introduce a config option (which will implicitly allow changes of the log level at runtime because the config is reloadable). This also implies to deprecate the command line flag, i.e. if you use it, you'll see a deprecation warning, and we will remove the flag in v4 (which might be years in the future).

The dev summit was very aware that there might be resistance against removing the command line option because it's very common to configure the log level via the command line. On the other hand, we also felt that breaking the well established patterns would also be very problematic (which is to have runtime changeable options in the config file and static options on the command line and never have two ways of configuring the same thing). So we settled on doing what's stated above, but of course, we can still reconsider in time before v4 is released.