search

aws-samples / amazon-cloudwatch-container-insights

CloudWatch Agent Dockerfile and K8s YAML templates for CloudWatch Container Insights.
MIT No Attribution
161 stars 106 forks source link

CloudWatch Prometheus agent can't read metrics export #34

Open rmmacala opened 4 years ago

rmmacala commented 4 years ago

The Cloudwatch prometheus agent is unable to read or determine the type of exported metric data:

2020-08-03T22:00:05Z D! [97/500] Unsupported Prometheus metric: kamailio_sl_200_replies with type:
2020-08-03T22:00:05Z D! [98/500] Unsupported Prometheus metric: kamailio_sl_202_replies with type:
2020-08-03T22:00:05Z D! [99/500] Unsupported Prometheus metric: kamailio_sl_2xx_replies with type:
2020-08-03T22:00:05Z D! [100/500] Unsupported Prometheus metric: kamailio_sl_300_replies with type:
2020-08-03T22:00:05Z D! [101/500] Unsupported Prometheus metric: kamailio_sl_301_replies with type:
2020-08-03T22:00:05Z D! [102/500] Unsupported Prometheus metric: kamailio_sl_302_replies with type:
2020-08-03T22:00:05Z D! [103/500] Unsupported Prometheus metric: kamailio_sl_3xx_replies with type:
2020-08-03T22:00:05Z D! [104/500] Unsupported Prometheus metric: kamailio_sl_400_replies with type:
2020-08-03T22:00:05Z D! [105/500] Unsupported Prometheus metric: kamailio_sl_401_replies with type:
2020-08-03T22:00:05Z D! [106/500] Unsupported Prometheus metric: kamailio_sl_403_replies with type:
2020-08-03T22:00:05Z D! [107/500] Unsupported Prometheus metric: kamailio_sl_404_replies with type:
2020-08-03T22:00:05Z D! [108/500] Unsupported Prometheus metric: kamailio_sl_407_replies with type:
2020-08-03T22:00:05Z D! [109/500] Unsupported Prometheus metric: kamailio_sl_408_replies with type:
2020-08-03T22:00:05Z D! [110/500] Unsupported Prometheus metric: kamailio_sl_483_replies with type:
2020-08-03T22:00:05Z D! [111/500] Unsupported Prometheus metric: kamailio_sl_4xx_replies with type:
2020-08-03T22:00:05Z D! [112/500] Unsupported Prometheus metric: kamailio_sl_500_replies with type:
2020-08-03T22:00:05Z D! [113/500] Unsupported Prometheus metric: kamailio_sl_5xx_replies with type:

The metric data is exported in counter format with no labels:

kamailio_sl_200_replies 12 1596492670513
kamailio_sl_202_replies 0 1596492670513
kamailio_sl_2xx_replies 0 1596492670513
kamailio_sl_300_replies 0 1596492670513
kamailio_sl_301_replies 0 1596492670513
kamailio_sl_302_replies 0 1596492670513
kamailio_sl_3xx_replies 0 1596492670513
kamailio_sl_400_replies 0 1596492670513
kamailio_sl_401_replies 0 1596492670513
kamailio_sl_403_replies 0 1596492670513
kamailio_sl_404_replies 0 1596492670513
kamailio_sl_407_replies 0 1596492670513
kamailio_sl_408_replies 0 1596492670513
kamailio_sl_483_replies 0 1596492670513
kamailio_sl_4xx_replies 0 1596492670513

The export data can be read by a stand alone Prometheus, I used the latest docker image prometheus:v2.19.3.

Is this export format not valid for the current version of the Cloud Watch Prometheus agent?

hdj630 commented 3 years ago

CloudWatch Agent needs to know the metric type to perform data processing accordingly, i.e. calculating the delta for counter metric. So a metric without type label is invalid to CloudWatch Agent.