open-telemetry / opentelemetry-proto-java

Java Bindings for the OpenTelemetry Protocol (OTLP)
https://opentelemetry.io
Apache License 2.0
15 stars 12 forks source link

Java pb parse data from collector error #8

Closed chenhaipeng closed 11 months ago

chenhaipeng commented 1 year ago

hi i found some error when i parse from bytes from opentelemetry collector, process error : tdlog.shade.com.google.protobuf.InvalidProtocolBufferException: Protocol message had invalid UTF-8. at tdlog.shade.com.google.protobuf.InvalidProtocolBufferException.invalidUtf8(InvalidProtocolBufferException.java:164) at tdlog.shade.com.google.protobuf.Utf8$UnsafeProcessor.decodeUtf8(Utf8.java:1419) at tdlog.shade.com.google.protobuf.Utf8.decodeUtf8(Utf8.java:340) at tdlog.shade.com.google.protobuf.CodedInputStream$ArrayDecoder.readStringRequireUtf8(CodedInputStream.java:815) at io.opentelemetry.proto.common.v1.AnyValue.(AnyValue.java:58) at io.opentelemetry.proto.common.v1.AnyValue.(AnyValue.java:15) at io.opentelemetry.proto.common.v1.AnyValue$1.parsePartialFrom(AnyValue.java:1468) at io.opentelemetry.proto.common.v1.AnyValue$1.parsePartialFrom(AnyValue.java:1462) at tdlog.shade.com.google.protobuf.CodedInputStream$ArrayDecoder.readMessage(CodedInputStream.java:889) at io.opentelemetry.proto.common.v1.KeyValue.(KeyValue.java:68) at io.opentelemetry.proto.common.v1.KeyValue.(KeyValue.java:14) at io.opentelemetry.proto.common.v1.KeyValue$1.parsePartialFrom(KeyValue.java:736) at io.opentelemetry.proto.common.v1.KeyValue$1.parsePartialFrom(KeyValue.java:730) at tdlog.shade.com.google.protobuf.CodedInputStream$ArrayDecoder.readMessage(CodedInputStream.java:889) at io.opentelemetry.proto.trace.v1.Span.(Span.java:121) at io.opentelemetry.proto.trace.v1.Span.(Span.java:20) at io.opentelemetry.proto.trace.v1.Span$1.parsePartialFrom(Span.java:6516) at io.opentelemetry.proto.trace.v1.Span$1.parsePartialFrom(Span.java:6510) at tdlog.shade.com.google.protobuf.CodedInputStream$ArrayDecoder.readMessage(CodedInputStream.java:889) at io.opentelemetry.proto.trace.v1.ScopeSpans.(ScopeSpans.java:77) at io.opentelemetry.proto.trace.v1.ScopeSpans.(ScopeSpans.java:13) at io.opentelemetry.proto.trace.v1.ScopeSpans$1.parsePartialFrom(ScopeSpans.java:1277) at io.opentelemetry.proto.trace.v1.ScopeSpans$1.parsePartialFrom(ScopeSpans.java:1271) at tdlog.shade.com.google.protobuf.CodedInputStream$ArrayDecoder.readMessage(CodedInputStream.java:889) at io.opentelemetry.proto.trace.v1.ResourceSpans.(ResourceSpans.java:78) at io.opentelemetry.proto.trace.v1.ResourceSpans.(ResourceSpans.java:13) at io.opentelemetry.proto.trace.v1.ResourceSpans$1.parsePartialFrom(ResourceSpans.java:2195) at io.opentelemetry.proto.trace.v1.ResourceSpans$1.parsePartialFrom(ResourceSpans.java:2189) at tdlog.shade.com.google.protobuf.CodedInputStream$ArrayDecoder.readMessage(CodedInputStream.java:889) at io.opentelemetry.proto.collector.trace.v1.ExportTraceServiceRequest.(ExportTraceServiceRequest.java:59) at io.opentelemetry.proto.collector.trace.v1.ExportTraceServiceRequest.(ExportTraceServiceRequest.java:9) at io.opentelemetry.proto.collector.trace.v1.ExportTraceServiceRequest$1.parsePartialFrom(ExportTraceServiceRequest.java:935) at io.opentelemetry.proto.collector.trace.v1.ExportTraceServiceRequest$1.parsePartialFrom(ExportTraceServiceRequest.java:929) at tdlog.shade.com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:158) at tdlog.shade.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:191) at tdlog.shade.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:203) at tdlog.shade.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:208) at tdlog.shade.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:48) at io.opentelemetry.proto.collector.trace.v1.ExportTraceServiceRequest.parseFrom(ExportTraceServiceRequest.java:268) at com.tencent.teg.tdlog.etl.processor.OpentelemetryTraceProcessor.processElement(OpentelemetryTraceProcessor.java:170) at com.tencent.teg.tdlog.etl.processor.OpentelemetryTraceProcessor.processElement(OpentelemetryTraceProcessor.java:61) at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:71) at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processElement(StreamOneInputProcessor.java:183) at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:156) at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:308) at org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:330) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:435) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:545) at java.lang.Thread.run(Thread.java:748)```

Cloud-Yao commented 11 months ago

@chenhaipeng 您好,我也正在尝试用Java解析来自Opentelemtry Collector的数据,也是解析出错,报错信息“Protocol message tag had invalid wire type”,想请教一下您。 这是我的Java代码

@PostMapping(value = "/v1/logs") public ExportLogsServiceResponse acceptData(@RequestBody byte[] data) throws IOException {

// System.out.println("字节" + bytes.length); // // 每次执行这个代码的时候就会报错,Protocol message tag had invalid wire type ExportLogsServiceRequest exportLogServiceRequest = ExportLogsServiceRequest.parseFrom(data);

    return null;
}

config: receivers: otlp: protocols: grpc: {} http: {} processors: batch: {} filter: {}

exporters: debug: {} logging: {} otlphttp: endpoint: http://192.168.18.5:8080/

extensions: health_check: {} memory_ballast: {}

service: pipelines: logs: receivers: [otlp] processors: [batch] exporters: [otlphttp]

这是我的Opentelemtry Collector配置

Cloud-Yao commented 11 months ago
image

Java代码

jkwatson commented 11 months ago

These bindings are provided as-is, generated by the protobuf compiler. We don't provide any more support beyond publishing them. If you need something custom, you are free to generate your own protobuf bindings from the proto definitions.

Cloud-Yao commented 11 months ago

@jkwatson hi, thank you for you answer. As so far,I don't define any protobuf file or data , i just use official provide helm chart deploy opentelemetry collector to send k8s logs to my java web endpoint, so i use official provide proto-java sdk to parse logs data ,but get error. I want to try accept opentelemetry data on java web code , process it and send my mysql database. And This is my question detail https://github.com/open-telemetry/opentelemetry-proto/issues/521

jkwatson commented 11 months ago

Unfortunately, this is not something we officially support. I think you will probably have to do some detailed debugging on your own.

chenhaipeng commented 11 months ago

@chenhaipeng 您好,我也正在尝试用Java解析来自Opentelemtry Collector的数据,也是解析出错,报错信息“Protocol message tag had invalid wire type”,想请教一下您。 这是我的Java代码

@PostMapping(value = "/v1/logs") public ExportLogsServiceResponse acceptData(@requestbody byte[] data) throws IOException {

// System.out.println("字节" + bytes.length); // // 每次执行这个代码的时候就会报错,Protocol message tag had invalid wire type ExportLogsServiceRequest exportLogServiceRequest = ExportLogsServiceRequest.parseFrom(data);

    return null;
}

config: receivers: otlp: protocols: grpc: {} http: {} processors: batch: {} filter: {}

exporters: debug: {} logging: {} otlphttp: endpoint: http://192.168.18.5:8080/

extensions: health_check: {} memory_ballast: {}

service: pipelines: logs: receivers: [otlp] processors: [batch] exporters: [otlphttp]

这是我的Opentelemtry Collector配置

I think you can run this example https://github.com/open-telemetry/opentelemetry-java-examples/blob/main/log-appender/README.md . Judging from the information you provided, your pb format is incorrect. You can try the official collector to print the log to see if the log type format is correctly parsed.

chenhaipeng commented 11 months ago

The above error is due to non-UTF-8 characters in attribute. The official go collector is compatible, but the java pb parsing tool is not compatible. I modified the java pb parsing tool to temporarily solve it.