Greenfield Logging/Alerting discussion

[msheiny]

Container based OS/green-field -- logging/alerting discussion

Feature request

Description

Since we effectively get a clean slate here to re-design the logging and alerting story, let's first break-down what we are trying to collect and when we think admins should be alerted.

A big problem with the current OSSEC design is that alerts are NOT action-able and thereby easily dismissed/neglected. We need to keep that in mind that alerts should only be sent in a situation where we expect an admin to look at the info and make a quick asssessment of whether they need to take action. Many many discussions in github also indicate we need to move away from email (lots of "hey lets move to signal" - like #1124 )

Might want to break this ticket up when implementation time comes to further discuss specific issues, but here we go...

Metric data we want to collect:

• Aggregated stats on uploads - want to avoid collecting meta-data that gives away source post/login times/dates
• Host Disk/CPU/Memory usage
• Tor network status
• Administrative/journalist login events - ssh/console/web
• System config change events -- new users added/removed, securedrop settings modified
• Container service status - when containers bounce, versions are updated, container host-level issues
• File manipulation - in containers and on host (currently using ossec to handle this)
• All other logs - from all containers, hosts

Basic features we want:

• Aggregation of multiple alerts -- lets keep alert blasts to a minimum
• Easy Upgrade Path -- painless for FPF to push updated rule-sets, filters, newer log system binaries
• Extensive filter and conditional logic for alert rules
• Push-notifications to cell phones -- signal is the top contender but we should support others as well (especially with the internationalization effort)
• Alerts to journalists on available submissions - see #1195, #544
• Ability to consolidate data from multiple sources (each container and host)

Nice to have (optional/reach-goals):

• Admin ability to filter alerts they'd like to receive
• UI Interface to scroll back and look at historical data
• Ability to export to compatible third party logging solutions (if an enterprise org wants to totally throw away our alerting logic and use something custom, we should support that by internally support piping out logs). Obvious risk here that we need to be careful not to leak meta-data about sources.
• Aggregated/anonymous opt-in ability for data export to FPF

When admin should be alerted:

• OS/Hardware issue -- disk space, overheat, high-load
• User activity -- user logs into the OS (ssh/terminal), new user edited/created/destroyed.
• File hash mismatch -- system files are not matching what FPFs expects to see (would require some kind of hash download from FPF for comparison)
• system updates -- lets keep this to app-code updates that FPF mandates -- we shouldn't be sending alerts when the base OS updates.
• system errors -- needs to be heavily tested and paired down to ensure we ignore any "regular" log activity, also include things like apparmor/selinux/grsec errors that pop up that might be signs of foul play

User Stories

As a securedrop administrator I would like sane action-able alerts

[ageis]

This ticket appears crazily ambitious, and I don't know if it's current. It sounds like you want a full-service DevOps/SRE solution.

The features in alerting that you were seeking are all covered by Prometheus Alertmanager. Alerts can be grouped, filtered, they're written with easy to understand conditional logic, etc. e.g.

  - alert: SecureDropInstanceDown
    expr: probe_http_status_code != 200
    for: 10m
    labels:
      severity: critical
    annotations:
      description: '{{ $labels.instance }} at {{ $labels.address }} might be down'
      summary: '{{ $labels.instance }} is not returning HTTP 200/OK'

Addressing a couple of other things you mentioned...

For monitoring the Tor network, I highly recommend: https://github.com/atx/prometheus-tor_exporter which I've used at Calyx.

For monitoring containers, see Cadvisor, or secondarily a product like Sysdig.

With regard to aggregated statistics about uploads, I did have that in https://github.com/freedomofpress/securedrop/pull/4414 which was understandably declined.