Prometheus - Exemplar label name is invalid when filtering attributes in a custom view

Description

After upgrading from go.opentelemetry.io/otel/exporters/prometheus v0.50.0 to go.opentelemetry.io/otel/exporters/prometheus v0.53.0 I see the following error exemplar label name \"wg.operation.hash\" is invalid.

We use a custom view to filter metric attributes with high cardinality. While the metric is definitely dropped in the view it seems the data is still exported on the prometheus exporter.

It is necessary to checkout https://github.com/open-telemetry/opentelemetry-go/commit/664a075380b9eb155512adb1213ca688bd1ed611 to not run into a different exemplar prometheus issue.

var opts []sdkmetric.Option

    // Exclude attributes from metrics

    attributeFilter := func(value attribute.KeyValue) bool {
        if isKeyInSlice(value.Key, defaultExcludedOtelKeys) {
            return false
        }
        name := sanitizeName(string(value.Key))
        for _, re := range c.Prometheus.ExcludeMetricLabels {
            if re.MatchString(name) {
                return false
            }
        }
        return true
    }

    msBucketHistogram := sdkmetric.AggregationExplicitBucketHistogram{
        Boundaries: msBucketsBounds,
    }
    bytesBucketHistogram := sdkmetric.AggregationExplicitBucketHistogram{
        Boundaries: bytesBucketBounds,
    }

    var view sdkmetric.View = func(i sdkmetric.Instrument) (sdkmetric.Stream, bool) {
        // In a custom View function, we need to explicitly copy the name, description, and unit.
        s := sdkmetric.Stream{Name: i.Name, Description: i.Description, Unit: i.Unit}

        // Use different histogram buckets for PrometheusConfig
        if i.Unit == unitBytes && i.Kind == sdkmetric.InstrumentKindHistogram {
            s.Aggregation = bytesBucketHistogram
        } else if i.Unit == unitMilliseconds && i.Kind == sdkmetric.InstrumentKindHistogram {
            s.Aggregation = msBucketHistogram
        }

        // Filter out metrics that match the excludeMetrics regexes
        for _, re := range c.Prometheus.ExcludeMetrics {
            promName := sanitizeName(i.Name)
            if re.MatchString(promName) {
                // Drop the metric
                s.Aggregation = sdkmetric.AggregationDrop{}
                return s, true
            }
        }

        // Filter out attributes that match the excludeMetricAttributes regexes
        s.AttributeFilter = attributeFilter

        return s, true
    }

    opts = append(opts, sdkmetric.WithView(view))

Stacktrace of the OTEL error

16:34:58 PM ERROR trace/meter.go:239 otel error {"hostname": "dustins-MacBook-Pro.local", "pid": 79745, "component": "@wundergraph/router", "service_version": "dev", "error": "exemplar label name \"wg.operation.hash\" is invalid"}
github.com/wundergraph/cosmo/router/pkg/trace.NewTracerProvider.func3
        /Users/starptech/p/wundergraph/cosmo/router/pkg/trace/meter.go:239
go.opentelemetry.io/otel.ErrorHandlerFunc.Handle
        /Users/starptech/go/pkg/mod/go.opentelemetry.io/otel@v1.31.1-0.20241030054014-4f94b1e661e7/error_handler.go:26
go.opentelemetry.io/otel.Handle
        /Users/starptech/go/pkg/mod/go.opentelemetry.io/otel@v1.31.1-0.20241030054014-4f94b1e661e7/handler.go:33
go.opentelemetry.io/otel/exporters/prometheus.addExemplars[...]
        /Users/starptech/go/pkg/mod/go.opentelemetry.io/otel/exporters/prometheus@v0.53.1-0.20241030054014-4f94b1e661e7/exporter.go:535
go.opentelemetry.io/otel/exporters/prometheus.addHistogramMetric[...]
        /Users/starptech/go/pkg/mod/go.opentelemetry.io/otel/exporters/prometheus@v0.53.1-0.20241030054014-4f94b1e661e7/exporter.go:260
go.opentelemetry.io/otel/exporters/prometheus.(*collector).Collect
        /Users/starptech/go/pkg/mod/go.opentelemetry.io/otel/exporters/prometheus@v0.53.1-0.20241030054014-4f94b1e661e7/exporter.go:229
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1
        /Users/starptech/go/pkg/mod/github.com/prometheus/client_golang@v1.20.5/prometheus/registry.go:458

Environment

OS: iOS
Architecture: ARM
Go Version: 1.21
opentelemetry-go version: 1.31.1

Expected behavior

It should be possible to filter attributes like before including prometheus exporter support

open-telemetry / opentelemetry-go