search

ballerina-platform / ballerina-lang

The Ballerina Programming Language
https://ballerina.io/
Apache License 2.0
3.68k stars 751 forks source link

[Bug]: Intermittent test failure in master for :jballerina-debugger-integration-test:test #41692

Open lochana-chathura opened 11 months ago

lochana-chathura commented 11 months ago

Description

$subject. Noticed :jballerina-debugger-integration-test:test failure in master quite frequently in recent times. Has to trigger the builds mutiple times to get the build passed.

Sample: https://github.com/ballerina-platform/ballerina-lang/actions/runs/6859271233/job/18651316246?pr=41683

1.

debugger-intg-test-suite > debugger-intg-tests > org.ballerinalang.debugger.test.adapter.run.MultiModuleRunDebugTest > testMultiModuleDebugScenarios FAILED
    org.ballerinalang.test.context.BallerinaTestException: Breakpoints request failed.
        at app//org.ballerinalang.debugger.test.utils.DebugTestRunner.setBreakpoints(DebugTestRunner.java:298)
        at app//org.ballerinalang.debugger.test.utils.DebugTestRunner.addBreakPoint(DebugTestRunner.java:272)
        at app//org.ballerinalang.debugger.test.adapter.run.MultiModuleRunDebugTest.testMultiModuleDebugScenarios(MultiModuleRunDebugTest.java:109)

        Caused by:
        java.util.concurrent.TimeoutException
Nov 14, 2023 5:19:59 AM org.eclipse.lsp4j.jsonrpc.RemoteEndpoint fallbackResponseError
SEVERE: Internal error: com.sun.jdi.VMDisconnectedException: connection is closed
java.util.concurrent.CompletionException: com.sun.jdi.VMDisconnectedException: connection is closed
    at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
    at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1770)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1760)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
    at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
Caused by: com.sun.jdi.VMDisconnectedException: connection is closed
    at jdk.jdi/com.sun.tools.jdi.TargetVM.send(TargetVM.java:299)
    at jdk.jdi/com.sun.tools.jdi.VirtualMachineImpl.sendToTarget(VirtualMachineImpl.java:1169)
    at jdk.jdi/com.sun.tools.jdi.PacketStream.send(PacketStream.java:77)

2.

 debugger-intg-test-suite > debugger-intg-tests > org.ballerinalang.debugger.test.adapter.BreakpointVerificationTest > testOnTheFlyBreakpointVerification FAILED
    org.ballerinalang.test.context.BallerinaTestException: Breakpoints request failed.
        at app//org.ballerinalang.debugger.test.utils.DebugTestRunner.setBreakpoints(DebugTestRunner.java:298)
        at app//org.ballerinalang.debugger.test.utils.DebugTestRunner.addBreakPoint(DebugTestRunner.java:272)
        at app//org.ballerinalang.debugger.test.adapter.BreakpointVerificationTest.testOnTheFlyBreakpointVerification(BreakpointVerificationTest.java:102)

        Caused by:
        java.util.concurrent.TimeoutException
            at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1960)
            at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2095)
            at org.ballerinalang.debugger.test.utils.client.DAPRequestManager.setBreakpoints(DAPRequestManager.java:90)
            at org.ballerinalang.debugger.test.utils.client.DAPRequestManager.setBreakpoints(DAPRequestManager.java:84)
            at org.ballerinalang.debugger.test.utils.DebugTestRunner.setBreakpoints(DebugTestRunner.java:295)
            ... 2 more

3.

 debugger-intg-test-suite > debugger-intg-tests > org.ballerinalang.debugger.test.adapter.completions.DebugCompletionTest > testDebugCompletions FAILED
    org.ballerinalang.test.context.BallerinaTestException: Error occurred when fetching completions
        at app//org.ballerinalang.debugger.test.utils.DebugTestRunner.fetchCompletions(DebugTestRunner.java:822)
        at app//org.ballerinalang.debugger.test.adapter.completions.DebugCompletionTest.testDebugCompletions(DebugCompletionTest.java:64)

        Caused by:
        java.util.concurrent.TimeoutException
            at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1960)
            at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2095)
            at org.ballerinalang.debugger.test.utils.client.DAPRequestManager.completions(DAPRequestManager.java:155)
            at org.ballerinalang.debugger.test.utils.client.DAPRequestManager.completions(DAPRequestManager.java:149)
            at org.ballerinalang.debugger.test.utils.DebugTestRunner.fetchCompletions(DebugTestRunner.java:817)
            ... 1 more
lochana-chathura commented 11 months ago

https://github.com/ballerina-platform/ballerina-lang/actions/runs/6861344997/job/18656859525?pr=41693 build resulted in 178 tests completed, 8 failed, 116 skipped for the debugger tests

NipunaRanasinghe commented 11 months ago

The aforementioned failures are not reproducible locally and, there are no memory leaks found in the test suite. Therefore most probably this can be a cleanup glitch in the previous build/test tasks, leaving GitHub runner VMs short on resources to run the integration tests. (Noticed similar failures in LS perf tests too, probably due to the same reason).

Therefore to address these issues and to reduce our PR build time, we're planning to move all the integration tests (debugger, Testerina, and JBalllerina) into a separate job, and to run them in parallel.

In the meantime we will disable the most frequently failing test cases (mentioned in the issue description) ftm, and will enable them once the test parallelization work is done.