search

grafana / grafana-opentelemetry-java

Grafana OpenTelemetry distribution for Java
Apache License 2.0
32 stars 6 forks source link

ERROR io.opentelemetry.exporter.internal.http.HttpExporter - Failed to export logs. The request could not be executed. Full error message: timeout-------logs error with grafana-otel-java-agent #742

Open Venture704 opened 4 days ago

Venture704 commented 4 days ago

In my kubernetes cluster while using grafana-opentelemetry-java-agent as init container in my springboot apps i am encounting this issue. Details

springboot version 2.7.3 jhipster version-793

full Error ---- [otel.javaagent 2024-10-11 08:13:14:920 +0000] [BatchLogRecordProcessor_WorkerThread-1] DEBUG io.opentelemetry.sdk.logs.export.BatchLogRecordProcessor$Worker - Exporter failed [otel.javaagent 2024-10-11 08:13:25:931 +0000] [OkHttp http://grafana-k8s-monitoring-grafana-agent.monitoring.svc.cluster.local:4318/...] ERROR io.opentelemetry.exporter.internal.http.HttpExporter - Failed to export logs. The request could not be executed. Full error message: timeout java.net.SocketTimeoutException: timeout at okio.SocketAsyncTimeout.newTimeoutException(JvmOkio.kt:146) at okio.AsyncTimeout.access$newTimeoutException(AsyncTimeout.kt:186) at okio.AsyncTimeout$source$1.read(AsyncTimeout.kt:390) at okio.RealBufferedSource.indexOf(RealBufferedSource.kt:436) at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:329) at okhttp3.internal.http1.HeadersReader.readLine(HeadersReader.kt:29) at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:180) at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:110) at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) at io.opentelemetry.exporter.sender.okhttp.internal.RetryInterceptor.intercept(RetryInterceptor.java:91) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:517) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.net.SocketException: Socket closed at java.net.SocketInputStream.read(SocketInputStream.java:204) at java.net.SocketInputStream.read(SocketInputStream.java:141) at okio.InputStreamSource.read(JvmOkio.kt:93) at okio.AsyncTimeout$source$1.read(AsyncTimeout.kt:153) ... 22 more [otel.javaagent 2024-10-11 08:13:25:932 +0000] [BatchLogRecordProcessor_WorkerThread-1] DEBUG io.opentelemetry.sdk.logs.export.BatchLogRecordProcessor$Worker - Exporter failed

zeitlinger commented 4 days ago

@maryliag can we use the otel tester?

maryliag commented 4 days ago

yes! @Venture704 we have a tool, called otel-checker that currently have a few checks, such as permissions, correct environment variables, exporter types, etc, that can help identify if there are any basic issues with your setup.

Are you able to install and run a Go binary in your cluster? If so, the steps can be found here: https://github.com/grafana/otel-checker