ministryofjustice / observability-platform

Observability Platform
https://observability-platform.service.justice.gov.uk/
MIT License
0 stars 1 forks source link

❇️ Central OpenSearch #6

Open jacobwoffenden opened 11 months ago

jacobwoffenden commented 11 months ago

User Story

As a Observability Platform product engineer I want to offer OpenSearch for publishing logs So that we have a viable alternative to shipping EKS logs to CloudWatch

User Type(s)

Observability Platform engineering and Observability Platform tenants

Value

This isn't based on any feedback from tenants, but I am concerned that shipping to CloudWatch Logs might not scale for Cloud Platform or Data Platform's EKS clusters.

Assumptions / Hypothesis / Questions / Unknowns

Hypothesis

If we... [do a thing] Then... [this will happ]

Proposal

Build a central OpenSearch cluster that uses IAM (sigv4) authentication to write data, similar to how we offer Amazon Managed Prometheus

Each tenant role can only write to specific index

We can then create a Grafana data source which is scoped to that index and uses data source permissions to control access

excalidraw of proposal

Unknowns

Potential pitfalls that could cause the story to expand beyond its original scope. Ideally this section will remain blank.

Definition of done

Reference

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/cloudwatch_limits_cwl.html

https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ac.html

https://docs.fluentbit.io/manual/pipeline/outputs/opensearch

How to write good user stories