search

craigatk / opentelemetry-gradle-plugin

Instrument your Gradle build with OpenTelemetry observability to find and eliminate bottlenecks in your build
Apache License 2.0
32 stars 9 forks source link

Connection reset by peer #169

Open mleegwt opened 1 month ago

mleegwt commented 1 month ago

I've followed https://github.com/craigatk/opentelemetry-gradle-plugin#jaeger to setup Jaeger to receive traces. When enabling this plugin on a (quite small) internal project using Gradle 8.8 and with or without parallel mode for execution I consistently encounter the following stacktrace (that does not occur without this plugin):

Unexpected exception thrown.
org.gradle.internal.remote.internal.MessageIOException: Could not write '/127.0.0.1:61099'.
        at org.gradle.internal.remote.internal.inet.SocketConnection.flush(SocketConnection.java:140)
        at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionDispatch.run(MessageHub.java:333)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
        at org.gradle.internal.concurrent.AbstractManagedExecutor$1.run(AbstractManagedExecutor.java:47)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.io.IOException: Connection reset by peer
        at java.base/sun.nio.ch.SocketDispatcher.write0(Native Method)
        at java.base/sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:54)
        at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:132)
        at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:76)
        at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:53)
        at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:532)
        at org.gradle.internal.remote.internal.inet.SocketConnection$SocketOutputStream.writeWithNonBlockingRetry(SocketConnection.java:279)
        at org.gradle.internal.remote.internal.inet.SocketConnection$SocketOutputStream.writeBufferToChannel(SocketConnection.java:267)
        at org.gradle.internal.remote.internal.inet.SocketConnection$SocketOutputStream.flush(SocketConnection.java:261)
        at org.gradle.internal.remote.internal.inet.SocketConnection.flush(SocketConnection.java:138)
        ... 6 more

This also fails the build! This is the most annoying part of this issue. I don't seen any pointers towards anything specific apart from that it consistently happens during test execution. Some of these tests are kind of slow (approximately 20 seconds, sometimes more). Could that have anything to do with it?

mleegwt commented 1 month ago

A new test seems to indicate that it's not as simple as that:

    @Test
    void testIssue() throws InterruptedException {
        Thread.sleep(20_000);
    }
craigatk commented 1 month ago

Sorry you're running into this issue!

I've been trying to reproduce this issue locally but I haven't been able to reproduce it. Are you able to reproduce it in a public project that I can use? If so, it gives a much better shot that I'll be able to find the cause and fix it :)