Filter unexpected log messages containing PII

User Story or Problem Statement

We should filter out messages containing PII before they are sent to Datadog, so they we can prevent unexpected PII from being leaked.

Description or Additional Context

Though we've blocked the CMS from inserting PII via known patterns, it is extremely difficult to identify and modify every place where PII might be written to log. Additionally, we cannot prevent editors from inserting PII as content.

The work here is to set up filters for the Datadog agent on the CMS servers that find PII patterns in logs and intercept them before they are sent to Datadog.

Steps for Implementation

Common patterns to scrub: https://docs.datadoghq.com/logs/guide/commonly-used-log-processing-rules/

Log filter implementation: https://docs.datadoghq.com/agent/logs/advanced_log_collection/?tab=configurationfile&site=gov#filter-logs

Note that what Datadog describes as 'log scrubbing' - replacing PII with a masked version - is not available in our Datadog instance. We must use 'filtering' i.e. excluding log messages that contain patterns.

If we are not able to determine what PII patterns OCTO is most concerned about, implement emails and we will iterate.

Acceptance Criteria

[ ] Log messages generated by the CMS that contain PII patterns are visible in the CMS's logs
[ ] These same log messages are not visible in Datadog.

department-of-veterans-affairs / va.gov-cms