Unexpected behaviour with =~ matcher

[jmichalek132]

Describe the bug

Having a query like this:

{resource.deployment.environment=~"prod" && resource.service.name="service_name" && span.http.route=~".*"}

I would expect the query to only return traces from our prod environment, but instead we receive traces from both prod and preprod. Not sure this is a bug but at least defiantly not something I expected coming from promql and logql.

To Reproduce Steps to reproduce the behavior:

1. Start Tempo (2.3.0)
2. Perform Operations (Read/Write/Others)

Expected behavior

I would expect the query to only return traces from our prod environment.

Environment:

• Infrastructure: Kubernetes (EKS)
• Deployment tool: helm (tempo-distributed)

Additional Context

The reason for using =~ matcher even with a single value that doesn't use regex is in Grafana dashboard we have set of variables used to drilldown, and we support setting multiple envs in that specific dashboard. So the actual value in the query is populated from the variable and can be a regex too.

[mdisibio]

Hi thanks for mentioning this. This is interesting because Loki's regex matching behavior is different for labels vs line contents: {app=~"foo"} |~ "foo" contains two different interpretations. Conditional is here. Labels are matched for equality while contents are matched for substrings. Tempo does not distinguish any data like this and only has the second behavior. Any change here needs some consideration. As you mentioned the case app=~"foo" is already covered by the equality operator app="foo".

As a work-around for Grafana variables can you try {resource.deployment.environment =~ "^$env$"? Replace $env with your actual variable name. ^ and $ will force a match against the entire contents and the same behavior as promql/logql label matchers. It will work for single value or multi-value (abc|def)

[jmichalek132]

Hi thanks for mentioning this. This is interesting because Loki's regex matching behavior is different for labels vs line contents: {app=~"foo"} |~ "foo" contains two different interpretations. Conditional is here. Labels are matched for equality while contents are matched for substrings. Tempo does not distinguish any data like this and only has the second behavior. Any change here needs some consideration. As you mentioned the case app=~"foo" is already covered by the equality operator app="foo".

As a work-around for Grafana variables can you try {resource.deployment.environment =~ "^$env$"? Replace $env with your actual variable name. ^ and $ will force a match against the entire contents and the same behavior as promql/logql label matchers. It will work for single value or multi-value (abc|def)

Thank you for the context, given what you said I am okay with closing the issue, it just might be nice to call out that specific behaviour in traceql docs.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity. Please apply keepalive label to exempt this Issue.