Data Quality Connector

[smithclay]

The purpose and use-cases of the new component

The data quality connector emits logs and metrics on telemetry that violates configured policies, including but not limited to:

• the absence of specific resource semantic conventions
• the cardinality of attributes on metrics
• The presence of resource attributes that are deprecated or not permitted

The goal of the connector is providing better data quality diagnostics so teams can identify regressions or use the emitted telemetry to help them migrate to semconv.

The primary end-user of this is operational teams managing central collector infrastructure for many internal development teams that emit metrics, logs, and spans. While there is some overlap with the need to perform data quality checks during development or as part of a CI/CD process, the main purpose of the connector is to proactively catch (and be alerted) data quality issues as they happen.

(Inspired by some KubeCon EU conversations with @mlunadia @tigrannajaryan and others)

Example configuration for the component

TBD — this needs further discussion. Ideally there is a (short, "strict mode"-type) default that enforces semconv vs requiring hundreds of lines of YAML to be usable.

Telemetry data types supported

Metrics, traces, and logs. Anything with resource attributes.

Is this a vendor-specific component?

• [ ] This is a vendor-specific component
• [ ] If this is a vendor-specific component, I am proposing to contribute and support it as a representative of the vendor.

Code Owner(s)

No response

Sponsor (optional)

No response

Additional context

Would like to discuss this in an upcoming collector SIG.

[smithclay]

One other open question: this has a lot in common with the count connector -- open to feedback from those maintainers if this makes more sense as an addition to that component.

[MadVikingGod]

I have been working on a tool outside the collector to measure instrumentation quality: https://github.com/MadVikingGod/otel-semconv-checker. I would be glad to help with a common set of tools to solve this.

[djaglowski]

@smithclay, cool idea.

Regarding overlap w/ the count connector - in my opinion the use cases here are distinct enough from the count connector that it would make more sense as a separate component. You could certainly accomplish some similar things with the count connector, but the config could be tricky at best. Also, as you indicated, this connector would emit logs, which I don't think the count connector should likely do in any case.

[NasAmin]

Cardinality can be an entirely separate connector for metrics.

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.