sysdig - time series over usage quote

BCDevOps / developer-experience

This repository is used to track all work for the BCGov Platform Services Team (This includes work for: 1. Platform Experience, 2. Developer Experience 3. Platform Operations/OCP 3)

Apache License 2.0

8 stars 17 forks source link

sysdig - time series over usage quote #4962

Open ShellyXueHan opened 2 months ago

ShellyXueHan commented 2 months ago

Describe the issue We are being charged monthly of metrics (time series) over usage. Need to discuss with both sysdig and our procurement team how to process those in the future.

Definition of done

[x] discussion on quote processing
[x] investigate into metrics burst events
[x] setup alerts for metrics over-usage
[ ] contract amendment

ShellyXueHan commented 1 month ago

Update:

The abnormal spikes were due to node ID changes causing duplicated metrics, issue has been fixed from sysdig and won't happen again.

However, we are not able to cap the normal usage increase, we will setup alerts to monitor the time series but that's it. Sysdig recommended a 20% increase of metrics to safely cover us for the rest of the fiscal year.

lukegonis commented 1 month ago

Currently being processed/amended by Vicky K