DopplerHQ / kubernetes-operator

Apache License 2.0
44 stars 18 forks source link

Set Loglevel #57

Open jhoelzel opened 7 months ago

jhoelzel commented 7 months ago

Dear Team,

how can i set the loglevel of the operator so my kibana is not spammed with info logs that do nothing for me?

Thanks in advance.

to put this into persepective, here are the last 24h for an active doppler system:

grafik

nmanoogian commented 7 months ago

Thanks for writing in about this, @jhoelzel!

We don't currently have any mechanisms for adjusting the log level of the operator. The logging library that we're using under the hood sends all logs to stderr by default, which we've had our eye on fixing. Is this the primary issue for you? Or would you prefer to simply disable all logs below a certain log level?

For Doppler internal reference, we're also tracking this issue under ENG-7553.

miskr-instructure commented 5 months ago

+1 to this.

Hundreds of thousands of useless log messages per day. I think in production it should be possible to set it to only log warnings and errors.

The only use of info logs is seeing as the secrets are being synced, but the status of when a secret was last synced should be exposed either on the DopplerSecret (as a status field?) or via a metrics API of the operator instead of logging it every time.

gooseops commented 5 months ago

+1 Would like to see environment variables or command arguments that we can set in the helm chart values.yaml file to move LOG_LEVEL to different levels as needed.

miskr-instructure commented 5 months ago

Bonus note: we should be able to mute rate limit warnings too, or they should be logged much less frequently. Even though they are warnings, if they happen once they'll keep reoccurring a lot and cause a lot of spam in the logs. It's not informative to see them dozens of times a minute.