search

aws / amazon-managed-service-for-prometheus-roadmap

Amazon Managed Service for Prometheus Public Roadmap
Other
40 stars 3 forks source link

Provide ability to wipe off data for AMP workspace/metrics #25

Open datsabk opened 1 year ago

datsabk commented 1 year ago

Use case 1

A Prometheus Counter metrics is an incremental metric that takes a new datapoint and adds/subtracts value to existing counter value. In scenarios where we wish to reset this counter, presently the only solution to use a new metrics altogether which calls for changes across dashboards and alerts.

Use case 2

In scenarios where due to an internal implementation error we end up pushing metrics which PII data in the tags. This data, when detected, needs to be wiped off immediately by wiping off tsdb data by duration/metric. In such scenarios, destroying workspace and losing on all metric data is not an option.

Use case 3

When we reach Quota limits - we presently get 400 Bad request errors for two hours as mentioned on AMP quota limits page

metrics that have reported data in the past 2 hours

Feature Request

Provide the ability to wipe off Prometheus Data on demand by metric or just a complete wipeoff by age of data. This will allow us to quickly unblock metric submission instead of losing on 2 hours of data altogether. Destroying and recreating workspace leads to a lot of changes which are not practical.

ampabhi-aws commented 1 year ago

We're investigating bringing support for the Prometheus Delete Timeseries api to Cortex and AMP - https://prometheus.io/docs/prometheus/latest/querying/api/#delete-series. Would this help solve this need?

ampabhi-aws commented 1 year ago

Duplicates: https://github.com/aws/amazon-managed-service-for-prometheus-roadmap/issues/16

jfharden commented 1 year ago

We're investigating bringing support for the Prometheus Delete Timeseries api to Cortex and AMP - https://prometheus.io/docs/prometheus/latest/querying/api/#delete-series. Would this help solve this need?

This would be exactly the right solution for me too, it would also ease the pain for developers moving to AMP from self-hosted (or other SaaS provider) since it's already known and well documented