[Logging][Meta] Enhancements, docs and fixes

[TinaHeiligers]

Logging plays an essential role in any software and Kibana is no different. Kibana's first channel for communicating any issues, deprecation warnings, general operational flow, and other similar information and, as such, is relied on heavily in the absence of any other observability or monitoring integration.

Since switching over to the new login system late in the 7 series, we haven't paid as much attention to the integral feature as we perhaps should have and this is starting to cause problems for end-users.

This issue serves to capture individual logging-related requests and to monitor progress towards them.

Bugs

• [ ] ~[Logging] Time-interval rotation triggering policy waits for a log entry to rotate #132405~ wont fix
• [ ] ~[Breaking change] Switch logging to ECS-JSON layout by default #114968~
  • moved to ops team for now

Enhancements

• [ ] [Logging] Add callback interface to logging methods for message serialization #129110
• [ ] [Logging] Add date-pattern file naming during log-rotation #132400
• [ ] [Logging] Add reverse numeric rolling strategy #132403
• [ ] Improving Elasticsearch response logs #119062
• [ ] Add request duration to the Elasticsearch response logs #124208
• [ ] Add filtering of elasticsearch query/deprecation logging #123789
• [ ] https://github.com/elastic/kibana/issues/96193
• [ ] https://github.com/elastic/kibana/issues/120337

Triage/Stale

• [ ] [elasticsearch/logging] Add slowlogs config option. #58086
• [ ] Kibana should log an incoming request information #96193
• [ ] Rename setting logging.appenders.rolling-file #103084
• [ ] [logging] request for more metrics from metrics.ops #98673
• [ ] [core.logging] Provide mechanism for configuring default appenders for specific loggers #92082
• [ ] Fix and add warning for duplicate logger context names #79514

Docs

• [x] Document max for rolling file appender numeric strategy. 8.x branches, 7.17 branch
• [x] [Logging] Legacy and new system configuration #129381
• [ ] Provide Logging guide for Kibana developers #96194

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[TinaHeiligers]

Most of the issues we have open are enhancements and docs requests. In particular, the rolling file appended docs aren't explaining the configuration and system very well and/or are outdated.

We need to pay special attention to configuration bounds, specifying the min and max allowed values in addition to the defaults.