Search by error tag does not work

[bocharovf]

Describe the bug Filter "error=true" does not show traces with errors.

To Reproduce Steps to reproduce the behavior:

1. Use clickhouse storage plugin
2. Run HotROD and produce some traces
3. In Jaeger UI select "Redis" service and find traces. Check that there are traces with errors in Redis service.
4. Add Tags filter "error=true"
5. See "No trace results. Try another query."

Expected behavior Traces with Redis error found

Screenshots

Version (please complete the following information):

• OS: windows
• Jaeger version: 1.27
• Deployment: docker-compose
• Clickhouse plugin version: 0.8
• Clickhouse version: yandex/clickhouse-server:21

What troubleshooting steps did you try? Tag filter works as expected with another tags (e.g. "param.driverID").
Error tag filter works with Elasticsearch plugin.

I use that docker-compose to compare ELK and Clickhouse storages.

[pavolloffay]

cc) @EinKrebs

[rlex]

Same issue. http.status_code=200 works fine, error=true returns empty results. Tested with jaeger hotrod.

[bocharovf]

Root cause: there could be more than one tag with the same name in tags.key / tags.value columns of jaeger_index_local table but we only check the first occurrence of it. This is because we also serialize events (logs) into tags.* column.

So we end up with multiple "error" tag:

client-uuid=534bc4dfd268070c
driver_id=T762908C
error=redis timeout
error=true
...
span.kind=client

And only first value could be found because we use filter tags.value[indexOf(tags.key, ?)] == ?

Seems there is no easy way to get all array indexes of values so it's hard to check all tag values.

Should we really support searching for events (log) attributes? We could either stop serialize them into tags columns or at least prefix with "event." to distinguish from boolean error tag.