TraceQL Query with Multiple Events

[aldenmoreton]

I would like to be able to filter my traces by two events that happen in the same span. It is already possible to reference two events, but only if they are in different spans. No combination of sub-queries with these events is able to return the desired result.

My Attempted Queries: (Among MANY others)

{ event:name = "event name 1" } && { event:name = "event name 2" }
{ event:name = "event name 1" } ~ { event:name = "event name 2" }
...

My Expected Output: List of Traces which have a span with both events in them.

Debugging: I believe that this type of query is currently impossible, and will require a new feature. To create a query that can perform this operation one of two things must be implemented:

1. Allow multiple events to be queried in a predictable way in the same spanset

   { event1:name = "event name 1" && event2:name = "event name 2" }

   This would not be perfered as there is no president for something like this in the Resource and Span scopes

2. Create Spanset Logical Operator for Self Reference (perform logical checks between two sub-queries of the same span)

   { event:name = "event name 1" } == { event:name = "event name 2" }

I understand that the Event scope for TraceQL is very new, but I would love to see this feature added soon.

[mdisibio]

Hi, thanks for the really thoughtful writeup. Agree, checking multiple events on the same span is a gap, and so is the next step which is multiple conditions on the same event (i.e. event:name=X && event.attr=Y). This overlaps with array handling, and it's been hard to pin down a syntax and design we are happy with. It also has had to been paused while we prioritize some other work. A couple recent PRs for reference: https://github.com/grafana/tempo/pull/3867 https://github.com/grafana/tempo/pull/4034

Tried to accomplish this too, and agree it doesn't seem possible. {event:name="..."} does check against all events for any match, and individually we can find either event:

{ event:name="event name 1"}
{ event:name="event name 2"}

But combining them doesn't:

{ event:name="event name 1" && event:name="event name 2"}

Need to dig in this, it is a bit unexpected to me.

Allow multiple events to be queried in a predictable way in the same spanset

Yes, I think the array syntax will come into play here, which should be good, but the team still is pinning this down.

perform logical checks between two sub-queries of the same span

Agree this is something we are missing. In this case it is more the intersection between spansets. I.e. the same span was present in both, versus "the results between {A} and {B} are identical". I think it is straightforward to implement, but the syntax == isn't ideal.