linkedin / rest.li

Rest.li is a REST+JSON framework for building robust, scalable service architectures using dynamic discovery and simple asynchronous APIs.
rest.li
Other
2.51k stars 546 forks source link

Increase time between rate-limited logging from DualReadLoadBalancingMonitor #987

Closed jefflai2 closed 8 months ago

jefflai2 commented 8 months ago

Logging UriProperties diffs can cause frequent and very large log lines at the warn level due to large entries.

Since these logs are still valuable at warn level, increasing time for rate-limited logger

the-coder-o commented 8 months ago
  1. Adjust Log Levels: If the logs at the warn level are too verbose, you might want to adjust the levels at which certain information is logged. For instance, only log the most critical differences at the warn level, while more detailed diff information could be logged at a lower level, such as info or debug.
  2. Rate Limiting: Implement or adjust the rate limiting for your logging to handle bursts of log entries without losing critical information.
  3. Log Compression: Look into ways of compressing the log entries. For instance, if the UriProperties diffs are similar, you could log a summary or the delta changes instead of the full diff.
  4. Log Rotation and Management: Ensure that your logging system is set up to rotate logs to prevent individual log files from becoming too large. This often involves archiving older logs and removing them after a certain period.
  5. Asynchronous Logging: If the logging process is affecting performance, consider using asynchronous logging mechanisms that can handle the logging process in a separate thread or process.
  6. Log Analysis Tools: Utilize log analysis tools to filter and search through the logs more efficiently. This can help in quickly identifying important warnings without having to sift through large volumes of log data.
  7. Review Logging Practices: Review your current logging practices to ensure that you are only logging what is necessary. Verbose logging can be useful for debugging, but it can also lead to performance issues and difficulties in log analysis.
jefflai2 commented 8 months ago

Merged changes in https://github.com/linkedin/rest.li/pull/985