Open gharris1727 opened 7 months ago
cobexer
commented
7 months ago Thank you for providing a valid report.
The issue is in the backlog of the relevant team and is prioritized by them.
Communication between the Gradle Daemon and a test executor may also avoid using a channel discoverable from outside the Gradle daemon entirely.
gharris1727
commented
5 months ago Hi @cobexer!
Can you share an estimate for when the team may address this issue? I want to know whether I should solicit an Apache Kafka contributor to contribute a fix, or pursue a fix myself.
Thanks!
cpholt
commented
2 months ago I think we're hitting this same issue without kafka. I just upgraded gradle from 7.4 to 8.5 and now every jenkins job fails with the same mystery IOException. Lots of our test follow the same model where a client will repeatedly connect to a server. I'm assuming that we're seeing the same thing were a client accidentally hits a gradle port.
Has any progress been made on this?
My next step will be to try newer version of gradle after 8.5.
UPDATE:
After A LOT of test runs and narrowing down the failing test, it turns out my problem was caused by a unit test triggering a call to Runtime.halt(). My previous approach was to set a system property in the test task in gradle to block calls to the method in question. For some reason this stopped working in gradle 8.
gradle.taskGraph.whenReady { taskGraph -> taskGraph.allTasks.each { task -> if (task.name == "test") { task.systemProperties = [testingnoexit:"1"] task.ignoreFailures = true task.afterSuite { desc, result -> if (desc.getParent() == null) { ext.testFailures += result.getFailedTestCount() //count failures } } testCheck.dependsOn(task) } } }
My new approach is to try Class.forName() on a JUnit class. if that succeeds, I assume I'm in a unit test and the halt() call is avoided.
gharris1727
commented
2 months ago @cpholt I think your situation is better captured in #3610 because it involves calling System.exit and Runtime.halt. This issue is specific to the network connection interference, and the gradle worker exiting is a symptom, rather than a cause.
Kafka normally has control over the JVM execution lifetime, and calls System.exit/Runtime.halt. In tests we have to mock out the exit procedures to prevent the situation you're describing.
cpholt
commented
2 months ago @gharris1727 You are absolutely correct. I had worked through the System.exit/Runtime.halt under gradle 7.4 years ago. When I moved to 8.5/8.8, i was seeing a failure which lead me here (incorrectly). Once I realized it was the exit/halt issue, I had it resolved (again) withing a few minutes. I hadn't seen the #3610 issue, but that's it exactly.
Current Behavior
A Gradle executor experiences this exception:
And the build fails due to this error:
Expected Behavior
The connection should be accepted, and then disconnected once the received data is shown to be invalid. The Gradle executor should continue running and accepting connections until a valid connection has been made. The Gradle executor could keep running indefinitely without a connection, or given a moderate (1min) timeout before the connection is abandoned completely.
Context (optional)
We use Gradle to build the Apache Kafka project, and received this bug report from one of our contributors.
We have numerous tests which appear to leak clients in this way, and it appears that the particulars of the Gradle protocol and the Kafka protocol happen to cause the Gradle end of the connection to exit ungracefully.
We are also working to reduce the number of leaked clients and network connections in our project to mitigate this failure, but it is very likely that there are downstream users of Kafka clients which could experience the same style of defect in their tests, and have similar build failures. And more generally, any sort of client could happen to connect to the Gradle executor and trigger this failure with a crafted response. This could be happening without Kafka in use at all, theoretically.
This appears similar to the descriptions of #27512, #23602, and #12566 but I thought to open this issue separately with my reproduction case if it turns out to be distinct.
Steps to Reproduce
Here is a reproducer: https://github.com/gharris1727/gradle-crash-reproducer . For a specific failed build, see https://github.com/gharris1727/gradle-crash-reproducer/actions/runs/7632481625/job/20792782276 .
The reproducer has 4 modules, structured such that the tests for
app-2start after the tests forapp-1are already running.app-1is continuously searching for ports which it can't bind() the address for, and connecting to them and then immediately closing the connection. This specific reproducer relies on MacOS for sequential port allocation in order to make it more reliable, but I think the same failure is possible on any platform.Here's the logs from the reproducer, in case the action becomes inaccessible:
Gradle version
8.5
Build scan URL (optional)
No response
Your Environment (optional)
No response