Open geemark opened 1 year ago
Thanks for opening this issue. A team member should give feedback soon. In the meantime, feel free to check out the contributing guidelines.
@geemark We have updated otel-collector with improved logging of errors. Please update SigNoz to v0.11.3 and try again. If a batch of spans fails then report back the logs to us.
Hey, I'm having this same error here: "clickhouse [Encode]: mismatched len of columns", do you have a solution for that? I'm using the latest version of SigNoz 0.11.4
@V1ct0rHM Can you please send over logs?
@makeavish yeah, here it is:
2022-12-07T15:26:19.576Z info service/service.go:106 Everything is ready. Begin running and processing data. 2022-12-07T19:38:21.797Z error clickhousetracesexporter/writer.go:122 Could not write a batch of spans to model table: {"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "batch": [{"durationNano":7197025090333,"startTimeUnixNano":1670434700114428871,"serviceName":"api-service-stage","kind":2,"errorEvent":{},"tagMap":{"http.method":"","http.target":"","service.name":"api-service-stage","telemetry.sdk.language":"go"},"traceModel":{"durationNano":7197025090333,"startTimeUnixNano":1670434700114428871,"serviceName":"api-service-stage","kind":2,"references":[{"refType":"CHILD_OF"}],"tagMap":{"http.method":"","http.target":"","service.name":"api-service-stage","telemetry.sdk.language":"go"}}}], "error": "clickhouse [Encode]: mismatched len of columns"} github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).writeBatch /src/exporter/clickhousetracesexporter/writer.go:122 github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter /src/exporter/clickhousetracesexporter/writer.go:101 2022-12-07T19:38:21.797Z error clickhousetracesexporter/writer.go:102 Could not write a batch of spans {"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "error": "clickhouse [Encode]: mismatched len of columns"} github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter /src/exporter/clickhousetracesexporter/writer.go:102
@V1ct0rHM As I can see in the write batch of Otel collector TraceID and SpanID is missing. Which lang/framework does your apps use? How are you instrumenting?
Is this issue solved? I still suffer from this when trying to receive log from my custom instrumentation tools. Here are the object:
@dangbb I am not able to reproduce this on latest SigNoz Otel Collector I tried unmarshalling your data and writing to clickhouse and it inserted fine.
Which language instrumentation are you trying and can you share sample app which reproduces this issue?
Bug description
OpenTelemetry data (.NET) does not get stored. otel-collector log output:
2022-10-31T12:24:56.624Z error clickhousetracesexporter/writer.go:101 Could not write a batch of spans {"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "error": "clickhouse [Encode]: mismatched len of columns"} github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter /src/exporter/clickhousetracesexporter/writer.go:101
2022-10-31T12:25:06.629Z error clickhousetracesexporter/writer.go:101 Could not write a batch of spans {"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "error": "clickhouse [Encode]: mismatched len of columns"} github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter /src/exporter/clickhousetracesexporter/writer.go:101
2022-10-31T12:25:16.630Z error clickhousetracesexporter/writer.go:101 Could not write a batch of spans {"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "error": "clickhouse [Encode]: mismatched len of columns"} github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter /src/exporter/clickhousetracesexporter/writer.go:101
The error is logged every 10 seconds, with no other information than the above.
Expected behavior
Data would get saved without error ;)
How to reproduce
This is a specific issue with one of four similar services, where only one fails. I am not sure which data causes this.
Version information
SigNoz installed via K8S Helm charts with default values. K8S cluster is self-hosted.
Additional context
I was directed here from slack #support (by Ankit Nayan) to create a debug build.
describe table signoz_traces.signoz_index_v2;
ββnameββββββββββββββββ¬βtypeβββββββββββββββββββββββββββββββββ¬βdefault_typeββ¬βdefault_expressionββ¬βcommentββ¬βcodec_expressionββ¬βttl_expressionββ β timestamp β DateTime64(9) β β β β DoubleDelta, LZ4 β β β traceID β FixedString(32) β β β β ZSTD(1) β β β spanID β String β β β β ZSTD(1) β β β parentSpanID β String β β β β ZSTD(1) β β β serviceName β LowCardinality(String) β β β β ZSTD(1) β β β name β LowCardinality(String) β β β β ZSTD(1) β β β kind β Int8 β β β β T64, ZSTD(1) β β β durationNano β UInt64 β β β β T64, ZSTD(1) β β β statusCode β Int16 β β β β T64, ZSTD(1) β β β externalHttpMethod β LowCardinality(String) β β β β ZSTD(1) β β β externalHttpUrl β LowCardinality(String) β β β β ZSTD(1) β β β component β LowCardinality(String) β β β β ZSTD(1) β β β dbSystem β LowCardinality(String) β β β β ZSTD(1) β β β dbName β LowCardinality(String) β β β β ZSTD(1) β β β dbOperation β LowCardinality(String) β β β β ZSTD(1) β β β peerService β LowCardinality(String) β β β β ZSTD(1) β β β events β Array(String) β β β β ZSTD(2) β β β httpMethod β LowCardinality(String) β β β β ZSTD(1) β β β httpUrl β LowCardinality(String) β β β β ZSTD(1) β β β httpCode β LowCardinality(String) β β β β ZSTD(1) β β β httpRoute β LowCardinality(String) β β β β ZSTD(1) β β β httpHost β LowCardinality(String) β β β β ZSTD(1) β β β msgSystem β LowCardinality(String) β β β β ZSTD(1) β β β msgOperation β LowCardinality(String) β β β β ZSTD(1) β β β hasError β Bool β β β β T64, ZSTD(1) β β β tagMap β Map(LowCardinality(String), String) β β β β ZSTD(1) β β β gRPCMethod β LowCardinality(String) β β β β ZSTD(1) β β β gRPCCode β LowCardinality(String) β β β β ZSTD(1) β β β rpcSystem β LowCardinality(String) β β β β ZSTD(1) β β β rpcService β LowCardinality(String) β β β β ZSTD(1) β β β rpcMethod β LowCardinality(String) β β β β ZSTD(1) β β β responseStatusCode β LowCardinality(String) β β β β ZSTD(1) β β ββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββ΄βββββββββββββββββββββ΄ββββββββββ΄βββββββββββββββββββ΄βββββββββββββββββ