Log retention, rotation, and long-term storage.

mmarcotte commented 4 years ago

Looks like most of the logs that we have in place are set to Never Expire. It would seem unnecessary for our purposes, and we could specify this with Terraform for our Log Groups. A shorter timeframe like two weeks would suffice for most use cases, and we can store snapshots or longer-term log data in a less accessible format.

Decisions

[ ] Where should logs be stored for auditing purposes? (There may be a wider Court-wide storage location).
[ ] How long should logs be stored for auditing purposes?
[ ] How long should logs be stored in an immediately-accessible format (Kibana, Cloudwatch)?

Tasks

[ ] Implement a longer-term log storage solution for rotated logs to live.
[ ] Adjust Cloudwatch Log Groups retention policies in Terraform.
[ ] Rotate data in Kibana using a solution like Curator.

Notes

From a security/compliance perspective, the guidance on how long logs should be stored is "long enough for your business need".

JessicaMarine commented 3 years ago

@mmarcotte There might be some information that we want to keep longer than 2 weeks (e.g., viewing history of document). Can we talk about broad classes of data being logged and make decisions at the class-level versus all or nothing.

adunkman commented 3 years ago

There are also auditing requirements here with logs, @lauraGgit should be in this convo too. We have a kickoff tomorrow to generate a to-log list and discuss retention.

mmarcotte commented 3 years ago

Thanks @adunkman!

I will reach out to @michael-mcvicker for some information to help lock in the decisions. As for the length of retention for immediately searchable information, I think two weeks would be sufficient. And I believe we can grab a backup of an index and use a local Elasticsearch/Kibana instance to perform investigations prior to that window.

mmarcotte commented 3 years ago

How to aggregate this? We just need to be thoughtful about this. <3

ustaxcourt / ef-cms