Allow enabling audit, error, slow logging on Elasticsearch/Opensearch domains

[markdboyd]

In order to provide useful debugging and security information, we want to allow enabling audit, error, and slow query logging on Elasticsearch broker plans.

A second, related piece of work after this is complete will be to give customers some self-service access to logs to improve their visibility into their brokered services.

---

Security considerations

Adding audit, error, slow query logs should improve our platform and customer awareness of issues and improve our ability to respond to them

Proposed implementation

One option would be to support the creation of a custom Cloudwatch group per log type (audit, error, slow) per customer. That way, we can provision Cloudwatch groups that are only accessible for the IAM user of the brokered Elasticsearch domain. Furthermore, then we ensure that the Cloudwatch group only contains logs for that customer.

Questions

• What should the retention period on these Cloudwatch logs be? 7 days?

To do

• [ ] Write an ADR of proposed implementation
• [ ] Implement the necessary changes

[pburkholder]

Removing squad-assurance for now.

[rbogle]

@JasonTheMain this one might be a good one to tackle too.

[JasonTheMain]

I found documentation on how to do this to a already running instance using cloudwatch. Do we want this to be set automatically somehow or is enabling it manually okay @markdboyd ?

[markdboyd]

@JasonTheMain I think we need to make sure the broker sets this up for new instances that are brokered. But then also update existing Elasticsearch instances to have this enabled

[markdboyd]

After considering the needs of some of our customers, we decided that we should have separate cloudwatch log groups to go along with each customer domain. By taking that approach, we would allow each customer to access the contents of just their logs and possibly even support shipping them to other destinations like brokered s3 buckets.

Also, we should probably put a very limited duration on Cloudwatch log group retention by default, like 3 days, since costs for Cloudwatch logs can be quite large especially if every Elasticsearch domain had these logs enabled.

Also, at this time, we're going to de-prioritize this work because it's not near completion and it's not as high of a priority as other planned work for this PI.