Exclude host, os_check_timestamp, security-group, and upper_...

[platform-support-slack-integration[bot]]

Submitted By: steve.albers Topic: Datadog Team: Tier 1 Suppport

Exclude host, os_check_timestamp, security-group, and upper_bound tags from indexing on openresty.nginx_http_request_duration_seconds.count (I will add details in ticket)

[platform-support-slack-integration[bot]]

Slack Thread Link: https://dsva.slack.com/archives/CBU0KDSB1/p1713794284643019

[va-albers]

We need to reduce our custom metrics costs on Datadog. Our #1 high indexed metric is openresty.nginx_http_request_duration_seconds.count - we can reduce our metric cost by reducing the number of indexed tags on this metric.

• I can do this update, but I'm assuming Platform may want to?
• I will provide supporting research in the comment below.
• You can make this change by opening the metric here, selecting the "Manage Tags" button, selecting the "Exclude tags..." button and choosing the following tags in the "Add some tags" box:
• os_check_timestamp
• security-group
• upper_bound
• host Removing these four indexes on tags reduces the number of ingested custom metrics from ~150,000 to ~7000 (we pay approximately $6 per 100 metrics per month).

[va-albers]

To determine this list I pulled the list of all dashboards and monitors that use this custom metric:

Dashboards

• Infrastructure alerts
• Platform Infrastructure
• Platform Tech Team 1 Load Testing Overview
• Rebecca's Dashboard Tue, Apr 11, 1:33:04 pm
• VSP - Application Request Breakdown

  Monitors

• Reverse proxy error rate high

None of these use the listed tag elements for queries.
If we make this change and find that it impacts a query that we were not aware of, or in the future we want to index one of these tags, we can choose to re-add it (it will start indexing from the point it was re-added).