As a Observability Platform product engineer
I want to offer OpenSearch for publishing logs
So that we have a viable alternative to shipping EKS logs to CloudWatch
User Type(s)
Observability Platform engineering and Observability Platform tenants
Value
This isn't based on any feedback from tenants, but I am concerned that shipping to CloudWatch Logs might not scale for Cloud Platform or Data Platform's EKS clusters.
Assumptions / Hypothesis / Questions / Unknowns
Hypothesis
If we... [do a thing]
Then... [this will happ]
Proposal
Build a central OpenSearch cluster that uses IAM (sigv4) authentication to write data, similar to how we offer Amazon Managed Prometheus
Each tenant role can only write to specific index
We can then create a Grafana data source which is scoped to that index and uses data source permissions to control access
Unknowns
Potential pitfalls that could cause the story to expand beyond its original scope. Ideally this section will remain blank.
User Story
As a Observability Platform product engineer I want to offer OpenSearch for publishing logs So that we have a viable alternative to shipping EKS logs to CloudWatch
User Type(s)
Observability Platform engineering and Observability Platform tenants
Value
This isn't based on any feedback from tenants, but I am concerned that shipping to CloudWatch Logs might not scale for Cloud Platform or Data Platform's EKS clusters.
Assumptions / Hypothesis / Questions / Unknowns
Hypothesis
If we... [do a thing] Then... [this will happ]
Proposal
Build a central OpenSearch cluster that uses IAM (sigv4) authentication to write data, similar to how we offer Amazon Managed Prometheus
Each tenant role can only write to specific index
We can then create a Grafana data source which is scoped to that index and uses data source permissions to control access
Unknowns
Potential pitfalls that could cause the story to expand beyond its original scope. Ideally this section will remain blank.
Definition of done
Reference
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/cloudwatch_limits_cwl.html
https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ac.html
https://docs.fluentbit.io/manual/pipeline/outputs/opensearch
How to write good user stories