Centralized logging

[FedericoCeratto]

Investigate the implementation of:

• Centralized logging with one logserver in each datacenter
• Secure log delivery using TLS
• Add metadata to Explorer, Explorer's Nginx, API, API's Nginx to correlate requests
• Log compression, if needed
• Log signing, if needed
• Backup
• Log analysis, monitoring and alerting

Expected benefits:

• Gather evidence around security incidents
• Speed up debugging by monitoring the rate of errors and warnings from our applications
• React faster to application errors by integrating alerts with the current monitoring platform

Configuring monitoring and alerting can be very time consuming: I suggest we weight the benefits and the effort required for different solutions.

[FedericoCeratto]

Journald has its own tools to send and receive logs and support TLS with client and server certificate pinning: https://packages.debian.org/sid/systemd-journal-remote

[hellais]

I have made a copy of the logs from the retired servers in /srv/jupyter/home/retired-server-logs on ams-jupyter.ooni.nu.

We may want to ingest these too once we figure out how we are going to do this.

[FedericoCeratto]

Few tips on journald. It can index and search on arbitrary key-value fields: man systemd.journal-fields

Show only filename, function name, line number and message from high priority logs from a given unit: sudo journalctl -u fastpath -o verbose -p0..3 --output-fields=CODE_FILE,CODE_FUNC,CODE_LINE,MESSAGE

Example output:

    CODE_FILE=/usr/lib/python3.7/dist-packages/fastpath/core.py
    MESSAGE='input'
            Traceback (most recent call last):
              File "/usr/lib/python3.7/dist-packages/fastpath/core.py", line 577, in msm_processor
                summary = score_measurement(measurement, matches)
              File "/usr/lib/python3/dist-packages/statsd/client.py", line 30, in _wrapped
                return_value = f(*args, **kwargs)
              File "/usr/lib/python3.7/dist-packages/fastpath/core.py", line 455, in score_measurement
                summary = {k: msm[k] for k in fields}
              File "/usr/lib/python3.7/dist-packages/fastpath/core.py", line 455, in <dictcomp>
                summary = {k: msm[k] for k in fields}
            KeyError: 'input'
    CODE_LINE=581
    CODE_FUNC=msm_processor

Filter messages from a function and show one entry per line: sudo journalctl -u fastpath CODE_FUNC=_fetch_measurement

Manually create a log with custom fields from Python:

journal.send("hi", MOOD="happy", MYCNT=3)```

[FedericoCeratto]

Implemented