Sysdig - PVC metrics not accurate

[ShellyXueHan]

Describe the issue Running something like

avg(kubelet_volume_stats_capacity_bytes{namespace="0bd5ad-prod"}) by (persistentvolumeclaim)

will return a slightly higher number than the actual PVC limit.

Definition of done

• [ ] identify why
• [ ] fix

[ShellyXueHan]

Update:

So switching to data type (instead of percentage type) sysdig gives a better value at 99.98GiB, compared to the actual 100Gi PVC. The value sysdig has matches with prometheus , where 107347968000 bytes = 99.98GiB. So we are good with this one.

But still need to follow up with another finding: https://github.com/BCDevOps/platform-services/issues/169

• [ ] Test out how multiple pod restarts affect the metrics accuracy compared to prometheus

[ShellyXueHan]

https://github.com/bcgov-c/platform-tools/blob/main/nagios/runner/project/roles/nagios/tasks/long-term-metrics.yaml#L529