Support for log streaming visualization

[louzadod]

Is your feature request related to a problem? Please describe. Sometimes, it is necessary to follow the logs in near real time. According to 12-factor principles, logs are stream of events and there are situations under investigation that demand a coordinated user-interaction followed by log analysis.

Describe the solution you'd like

Given the context above, it is necessary some way, inside Dashboards UI, to follow the logs in realtime, Ideally, the fields to be shown should be customizable and the layout should be dev-oriented (fixed fonts, cool formatting to easy log reading).

There is an old open-source kibana plugin called LogTrail. Kibana version 7.10 changed its plugin API and this plugin stopped working. Anyway, I think it's approach, UI and layout can work as inspiration.

I see three possibilities here:

1. Adapt existing LogTrail plugin

2. Write a new plugin inspired on LogTrail

3. Design and implement a complete different plugin

Additional context

[tmarkley]

@louzadod thanks for submitting this idea! Have you investigated the three possibilities you listed? Is there one option that you're recommending? Are you asking for support from the core Dashboards team?

[louzadod]

Yes, @tmarkley . I'm asking for support. I want to start a discussion around this feature.

[ahopp]

@louzadod following up on @tmarkley's question - have you done any investigation on these options? OR is there 1 option you're recommending?

Any additional context you can provide to drive the discussion would be helpful!

[louzadod]

Sorry for the delay. Currently the developers at my organization gave me some feedback based on our existing Kibana/Logtrail on premises solution:

• requirement: follow the logs automatically (a la "tail -f")
• requirement: flexible filtering using queries and search on indexed fields (log message, por example)
• requirement: high performance on modern browsers.
• nice to have: ability to customize which fields are presented and the output format.

The general feeling of my colleagues is that it's a mission critical functionality, specially when you are under pressure investigating problems. Due to failures in the existing tooling, people want to have access to log files. My goal is to avoid this scenario.

[mrliptontea]

Came to this repo to see if something like this is already on the radar, I would definitely love to see a specialised view dedicated to logs. I know Kibana has Elastic Logs available in X-Pack.

To the above requirements I would add a few ideas:

• Saved searches - we use them to quickly look for logs for a specific app, with different fields visible depending on the logs format.
• Reverse order - basically to see logs appended to the bottom rather than the top, just like tail -f.
• Highlight line based on log level - e.g. Grafana Logs panel does this.
• Quick filters - just to filter what's already loaded on-screen (hide non-matching lines), without searching all over again. Useful when you want to toggle quickly between different filters. Should work nicely with tail -f mode.
• Toggle line wrap - log lines tend to be quite long, but the most useful bit is usually at the beginning. This basically could toggle a CSS class to let you see more lines on a single screen.

Just writing this down as I have yet to find a solution that's as convenient and flexible as tail on the terminal but without giving anyone access to the servers.