Hostmetrics Load Scraper does not have even 1 Label - `awsemf` exporter filtering depends on 1+ labels

[NathanielRN]

Is your feature request related to a problem? Please describe.

The hostmetrics cpu scraper will produce a metric like this (using the file exporter):

"metrics": [
    {
        "name": "system.cpu.time",
        "description": "Total CPU seconds broken down by different states.",
        "unit": "s",
        "sum": {
            "dataPoints": [
                {
                    "labels": [
                        {
                            "key": "cpu",
                            "value": "cpu0"
                        },
                        {
                            "key": "state",
                            "value": "user"
                        }
                    ],
                    "startTimeUnixNano": "1628655956000000000",
                    "timeUnixNano": "1628703962211403000",
                    "asDouble": 2255.98
                },
                etc...
            ],
            "aggregationTemporality": "AGGREGATION_TEMPORALITY_CUMULATIVE",
            "isMonotonic": true
        }
    }
]

but the hostmetrics load scraper will produce a metric like this:

"metrics": [
    {
        "name": "system.cpu.load_average.1m",
        "description": "Average CPU Load over 1 minute.",
        "unit": "1",
        "gauge": {
            "dataPoints": [
                {
                    "timeUnixNano": "1628703962211891000",
                    "asDouble": 1.958984375
                }
            ]
        }
    }
]

It has no labels. Without this label, the awsemf exporter will produce the follow output for the CloudWatch (CW) backend for the cpu metric:

{
    "_aws": {
        "CloudWatchMetrics": [
            {
                "Namespace": "LocalOnlyLoadNamesapce",
                "Dimensions": [
                    [
                        "cpu",
                        "state"
                    ]
                ],
                "Metrics": [
                    {
                        "Name": "system.cpu.time",
                        "Unit": "Seconds"
                    }
                ]
            }
        ],
        "Timestamp": 1628991083218
    },
    "cpu": "cpu0",
    "state": "interrupt",
    "system.cpu.time": 0
}

but will produce this for load

{
    "_aws": {
        "CloudWatchMetrics": [
            {
                "Namespace": "LocalOnlyLoadNamesapce",
                "Dimensions": [
                    []
                ],
                "Metrics": [
                    {
                        "Name": "system.cpu.load_average.15m",
                        "Unit": "1"
                    },
                    {
                        "Name": "system.cpu.load_average.1m",
                        "Unit": "1"
                    },
                    {
                        "Name": "system.cpu.load_average.5m",
                        "Unit": "1"
                    }
                ]
            }
        ],
        "Timestamp": 1628991854480
    },
    "system.cpu.load_average.15m": 0.39,
    "system.cpu.load_average.1m": 1.21,
    "system.cpu.load_average.5m": 0.72
}

While both of these are valid CW metrics, the former would show up as Metrics with no Dimensions in the AWS CW backend.

Then, if we want to use the feature from the awsemf exporter that allows you to filter the metrics it sends by their dimensions, we can't do it. Because without even one dimension, we can't do the following:

metric_declarations:
  # CPU
  - dimensions: [[state]]
    metric_name_selectors:
      - system\.cpu\.time
  # Load
  # FIXME: No dimensions so cannot be filtered
  - dimensions: [[]]
    metric_name_selectors:
      - system.cpu\.load_average\.1m

Describe the solution you'd like

If it makes sense for exporters to depend on these labels, we could update the hostmetrics load_scraper.go

https://github.com/open-telemetry/opentelemetry-collector/blob/ffda9d5b948bcac77e8b66d94962b2da68da82ba/receiver/hostmetricsreceiver/internal/scraper/loadscraper/load_scraper.go#L74-L79

to add at least 1 label:

labelsMap := dp.LabelsMap()
labelsMap.Insert("load", "load_1m")

which the cpu_scarper.go already does:

https://github.com/open-telemetry/opentelemetry-collector/blob/ffda9d5b948bcac77e8b66d94962b2da68da82ba/receiver/hostmetricsreceiver/internal/scraper/cpuscraper/cpu_scraper.go#L82-L93

Describe alternatives you've considered

Maybe the awsemfexporter should add a "default" dimension to every metric it receives. So for the cpu metric, in addition to [cpu, state] which correspond to (cpu0, system), it could also add a name dimension so that [cpu, state, name] correspond to (cpu0, system, system.cpu.time)?

That way load would have at least [name] which corresponds to (system.cpu.load_average.1m)

Additional context No additional context.

[mxiamxia]

FIXME: No dimensions so cannot be filtered

• dimensions: [[]] metric_name_selectors:
  • system.cpu.load_average.1m

This is a bug will be fixed in emfexporter. :)

Maybe the awsemfexporter should add a "default" dimension to every metric it receives.

We can't add this dimension to metrics. It's an extra cost to users.

[sethAmazon]

@alolita I would like to look into this bug

[sethAmazon]

@NathanielRN are you saying that no metrics without dimensions are being published to cw? From code https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/awsemfexporter/metric_translator.go#L239-L339 I think that is happening.

[NathanielRN]

@sethAmazon No sorry. From the issue description above I explain that although yes metrics without dimensions are normally being published to CW, if I use the awsemfs's metric filter (to include) metrics without dimensions they won't appear in CW. I explain in the issue that this is because the exporter looks at the labels to decide if the filter finds a match, but without labels the filter cannot match anything. So there needs to be a way to include metrics without dimensions in the awsemf exporter filter.

[lorelei-rupp-imprivata]

I think this is related to the same issue I am facing as well. We have a bunch of metrics that have no labels/dimensions that we cannot seem to get into Cloud Watch either. I linked my ticket, trying to get confirmation if it is indeed the same issue. Is there any update on this? Any potential ETA?

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[github-actions[bot]]

This issue has been closed as inactive because it has been stale for 120 days with no activity.