[FEATURE] [SSH Logs] Adapt for usage with systemd

nicolashimmelmann commented 1 month ago

Dashboard Improvement Suggestion Template

Dashboard Name: SSH Logs

Improvement Details

Description: Starting with Debian 12, SSH logs are written to a systemd journal, and can be viewed with journalctl -u ssh. The log file in /var/log/auth.log does not exist anymore.

I already adapted the Promtail configuration from your tutorial (see below), but I don't get any results from the Dashboard queries.

Use Case: Allow monitoring the SSH logs of any system that uses systemd journal for ssh logs

Proposed Changes

Change Description:

Update the tutorial to show a Promtail configuration to read SSH logs from systemd journal:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
- url: http://<loki-url>:3100/loki/api/v1/push

scrape_configs:
- job_name: systemd_journal_ssh
  journal:
    path: /var/log/journal
    matches: _SYSTEMD_UNIT=ssh.service
    labels:
      instance: my.instance.com
      env: prod
      job: ssh
  relabel_configs:
    - source_labels: ['__journal__systemd_unit']
      target_label: 'unit'

Reading a line like this from the journal:

Oct 29 14:49:47 proxmox sshd[549136]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)

results in the following line in Loki:

2024-10-29 14:49:47.846 pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)

So somehow the queries need to be changed, because they try to match |="sshd[", which is not contained in the resulting line. Or do I just need to change something in the Promtail config to still receive the full line in Loki? I am quite new to this.

Example of a current query: sum by(instance) (count_over_time({$label_name=~"$label_value", job=~"$job", instance=~"$instance"} |="sshd[" |=": session opened for" | __error__="" [$__interval]))

Rationale

Why is this Improvement Important: journalctl is the default for ssh logs in Debian starting with version 12

Checklist

Please ensure that you have completed the following tasks before submitting your improvement suggestion:

[x] Checked existing dashboards and features to confirm that the improvement is not already available.
[x] Reviewed any related documentation or resources to ensure your suggestion aligns with best practices.
[x] Considered the impact of your suggestion on the overall user experience.

Thank you for taking the time to suggest improvements to our Grafana dashboards! Your feedback is valuable in helping us enhance the usability and effectiveness of our monitoring and visualization tools.

voidquark commented 3 weeks ago

The first concern I see is that you’re using Debian, while I have only tested and officially support RedHat-based systems at this time. If you decide to proceed with Debian, you'll need to fork the dashboard and adjust it for compatibility with your Debian environment.

Additionally, the dashboard is built to expect log file and not systemd directly. So for example for RedHat you can use /var/log/secure and it parses sshd specific log entries.

In your case, you can try to remove |="sshd[" from all panels.

nicolashimmelmann commented 3 weeks ago

Thanks for your reply! I did not see RedHat mentioned anywhere in the repository, so sorry if I assumed incorrectly that Debian is supported. I removed the sshd part from the queries, and for now it works for me. Feel free to close the issue if there are no plans to support systemd :)

voidquark commented 3 weeks ago

It’s only mentioned here: Grafana SSH Logs Dashboard, but I’m glad it’s working now.

voidquark / grafana-dashboards