Drain logs from cloud.gov spaces to AWS Elasticsearch instance

[mogul]

User Story

In order to support data.gov's needs for monitoring and alerting on app behavior, the data.gov team wants to drain logs from cloud.gov spaces to a brokered instance of AWS ES.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• [x] GIVEN I am logged into the Kibana associated with the gsa-datagov/management space \ WHEN I generate traffic to a data.gov application in any space in the gsa-datagov organization \ THEN I see the activity logged via the Kibana UI.

Background

[Any helpful contextual notes or links to artifacts/evidence, if needed]

Security Considerations (required)

This story configures cloud.gov's platform-provided logging facilities to send logs from our spaces to a cloud.gov-brokered instance of AWS Elasticsearch. We assume that cloud.gov has ensured that the communication mechanisms are secured with TLS, and that client credentials are used for accessing the AWS ES endpoints; this should be verified. If IP restrictions are in place (eg only the cloud.gov egress IP ranges are permitted), then we should be noting that and documenting how to access the Kibana UI in our runbook. (Even if cloud.gov does not apply IP restrictions on its own, we should consider whether we want to add them ourselves.)

Note that AWS ES should be configured to prevent modifications of records in accordance with AU-9 if possible. If that's not possible, we may want to investigate how to also drain logs to CloudWatch Logs; this would likely happen via a separate issue.

Sketch

• [x] Broker an es-medium-ha instance of the aws-elasticsearch service in the management space
• [x] Try to figure out how cloud.gov is configuring the access control, and see if we need to do more (as described in the AWS docs for provisioning an instance).
• [x] Generate a syslog URL if necessary (meaning if cloud.gov and AWS ES do not supply one by default).
  • [ ] ~There's a syslog input plugin for ES.~ AWS ES doesn't allow installation of plugins.
  • [x] Alternatively, we can run fluent-bit as an app using the apt-buildpack, configured with the syslog input and the Elasticsearch output. (Cloud Foundry documents how to configure the input plugin.)
  • [x] Another option is using Logstash. Swisscom has documented how to do this using the stock docker image with Cloud Foundry.
• [x] Configure cloud.gov to drain logs to the service instance. Rather than binding each individual app in each space to the log drain, set up the log drain once for each space using the log-drain CF CLI plugin.
• [x] Verify that we can authenticate with Kibana and view the UI
• [x] Test that any logs generated are showing up in Elasticsearch.

[mogul]

Need to make a decision on Fluent Bit vs LogStash

[adborden]

notes to self:

Take a look at the logstash configs available from https://github.com/cloudfoundry-community/logsearch-for-cloudfoundry

How do we push our indexing config?

[adborden]

Memory calculation for Logstash https://github.com/cloudfoundry/java-buildpack-memory-calculator

[mogul]

And now, Brand X. https://sematext.com/docs/logagent/ https://sematext.com/docs/logagent/input-plugin-cloudfoundry/ https://sematext.com/docs/logagent/output-plugin-aws-elasticsearch/