Long urls in the alerts

[ankitnayan]

Another thing that we encountered is long SigNoz url's. We have log based alerting configured to our slack channel which provides a link to the relevant query. The query can become big but the generated url even bigger to a point that the url is to big for a slack message. This could be just an "us" issue with to big queries, but we've never had this issue with our previous log tooling.

Slack Message

[ngudbhav]

We are also facing this issue. The homepage graphs like Latency, Apdex and Throughput do not load resulting in 400. Digging through the logs, It says Maximum query size exceeded.

"Max query size exceeded: ","stacktrace":"go.signoz.io/signoz/pkg/query-service/app/clickhouseReader.(*ClickHouseReader).GetTimeSeriesResultV3\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/clickhouseReader/reader.go:4646\ngo.signoz.io/signoz/pkg/query-service/app/querier/v2.(*querier).execClickHouseQuery\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/querier/v2/querier.go:95\ngo.signoz.io/signoz/pkg/query-service/app/querier/v2.(*querier).runBuilderQuery\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/querier/v2/helper.go:281"}

I added the below clickhouse config

<max_query_size>1000000</max_query_size>

The graphs are now working but the query alone is taking about 10s

[srikanthccv]

It's not the same thing. Please create a separate issue. Make sure your instrumentation follows the naming convention as mentioned here https://opentelemetry.io/docs/specs/otel/trace/api/#span.

The span name concisely identifies the work represented by the Span, for example, an RPC method name, a function name, or the name of a subtask or stage within a larger computation. The span name SHOULD be the most general string that identifies a (statistically) interesting class of Spans, rather than individual Span instances while still being human-readable. That is, “get_user” is a reasonable name, while “get_user/314159”, where “314159” is a user ID, is not a good name due to its high cardinality. Generality SHOULD be prioritized over human-readability.

[srikanthccv]

@DanikRaikhlin you mentioned this in Slack. The long URLs have been changed to hyperlinks with text here https://github.com/SigNoz/signoz/pull/5001/files#diff-5d93d2488907ea7ecfb28b45545142baa71a66a2999cf4420e9385a52ff8ddc4. As you have already created the notification channel, it wouldn't reflect. Please edit the channel and replace the

     *RelatedLogs:* {{ .Annotations.related_logs }}
     *RelatedTraces:* {{ .Annotations.related_traces }}

with

     *RelatedLogs:* {{ if gt (len .Annotations.related_logs) 0 -}} View in <{{ .Annotations.related_logs }}|logs explorer> {{- end}}
     *RelatedTraces:* {{ if gt (len .Annotations.related_traces) 0 -}} View in <{{ .Annotations.related_traces }}|traces explorer> {{- end}}

And check again.

[ankitnayan]

@srikanthccv should these be included by default?

     *RelatedLogs:* {{ if gt (len .Annotations.related_logs) 0 -}} View in <{{ .Annotations.related_logs }}|logs explorer> {{- end}}
     *RelatedTraces:* {{ if gt (len .Annotations.related_traces) 0 -}} View in <{{ .Annotations.related_traces }}|traces explorer> {{- end}}

[srikanthccv]

They have been included by default since the last release. Those who used the old version of SigNoz will not have these annotations because the template would have already been saved.

[DanikRaikhlin]

@srikanthccv Thanks that indeed worked!