pkg/dds/test/dap/integration/debug_exceptions_test went from flaky -> RuntimeError on unittest-asserts-release-win-arm64

[derekxu16]

The tests

pkg/dds/test/dap/integration/debug_exceptions_test RuntimeError (expected Pass)

are failing on configurations

unittest-asserts-release-win-arm64

https://dart-ci.appspot.com/log/pkg-win-release-arm64/unittest-asserts-release-win-arm64/1503/pkg/dds/test/dap/integration/debug_exceptions_test

https://ci.chromium.org/ui/p/dart/builders/ci.sandbox/pkg-win-release-arm64/1503/overview

I see two errors in the logs. The first is a SocketException:

00:02 +0 -1: debug mode does not pause on exceptions if mode not set [E]

  SocketException: Write failed (OS Error: The pipe is being closed.

  , errno = 232), port = 0

  dart:io-patch/socket_patch.dart 1246:34       _NativeSocket.write
  dart:io-patch/socket_patch.dart 2004:15       _RawSocket.write
  dart:io-patch/socket_patch.dart 2481:18       _Socket._write
  dart:io-patch/socket_patch.dart 2216:28       _SocketStreamConsumer.write
  dart:io-patch/socket_patch.dart 2168:11       _SocketStreamConsumer.addStream.<fn>
  dart:async/zone.dart 1407:47                  _rootRunUnary
  dart:async/zone.dart 1308:19                  _CustomZone.runUnary
  dart:async/zone.dart 1217:7                   _CustomZone.runUnaryGuarded
  dart:async/stream_impl.dart 365:11            _BufferingStreamSubscription._sendData
  dart:async/stream_impl.dart 541:14            _DelayedData.perform
  dart:async/stream_impl.dart 646:11            _PendingEvents.handleNext
  dart:async/stream_impl.dart 617:7             _PendingEvents.schedule.<fn>
  dart:async/zone.dart 1391:47                  _rootRun
  dart:async/zone.dart 1301:19                  _CustomZone.run
  dart:async/zone.dart 1209:7                   _CustomZone.runGuarded
  dart:async/zone.dart 1249:23                  _CustomZone.bindCallbackGuarded.<fn>
  dart:async/zone.dart 1399:13                  _rootRun
  dart:async/zone.dart 1301:19                  _CustomZone.run
  dart:async/zone.dart 1209:7                   _CustomZone.runGuarded
  dart:async/zone.dart 1249:23                  _CustomZone.bindCallbackGuarded.<fn>
  dart:async/schedule_microtask.dart 40:21      _microtaskLoop
  dart:async/schedule_microtask.dart 49:5       _startMicrotaskLoop
  dart:isolate-patch/isolate_patch.dart 118:13  _runPendingImmediateCallback
  dart:isolate-patch/isolate_patch.dart 185:5   _RawReceivePort._handleMessage

The second is a drive letter capitalization mismatch:

00:14 +5 -1: debug mode pauses on uncaught exceptions when mode=Unhandled (supportUris: false, sendFileUris: false)

Request "terminate" has taken longer than 10s

00:14 +5 -2: debug mode parses line/column information from stack traces [E]

  Expected: 'c:\\opt\\s\\w\\ir\\x\\t\\dart-sdk-dap-test5e2fb1cf\\app19a5c4a8\\test_file.dart'
    Actual: 'C:\\opt\\s\\w\\ir\\x\\t\\dart-sdk-dap-test5e2fb1cf\\app19a5c4a8\\test_file.dart'
     Which: is different.
            Expected: c:\\opt\\s ...
              Actual: C:\\opt\\s ...
                      ^
             Differ at offset 0

  package:matcher/src/expect/expect.dart 149:31                 fail
  package:matcher/src/expect/expect.dart 144:3                  _expect
  package:matcher/src/expect/expect.dart 56:3                   expect
  pkg\dds\test\dap\integration\debug_exceptions_test.dart 91:7  main.<fn>.<fn>
  ===== asynchronous gap ===========================
  package:test_api/src/backend/declarer.dart 215:9              Declarer.test.<fn>.<fn>
  ===== asynchronous gap ===========================
  package:test_api/src/backend/declarer.dart 213:7              Declarer.test.<fn>
  ===== asynchronous gap ===========================
  package:test_api/src/backend/invoker.dart 258:9               Invoker._waitForOutstandingCallbacks.<fn>

[derekxu16]

cc @bkonyi and @DanTup

[DanTup]

https://dart-review.googlesource.com/c/sdk/+/365804 should fix the drive letter mismatch (oddly that test hasn't changed in 2yrs.. not sure if the drive letter casing is somehow different on some bots for temp directory?).

Hard to tell what the other failure is - @derekxu16 is there a way to tell how frequently it's failing? We could enable verbose output on the bots temporarily to try to get a better idea of what's happening?

[derekxu16]

We have a flakiness dashboard. Unfortunately you have to log in with an @google.com account to access it, but here's a screenshot showing the run results of pkg/dds/test/dap/integration/debug_exceptions_test on Windows:

I don't know how many of these failures were caused by SocketExceptions specifically, but the test is failing frequently.

[DanTup]

I guess many of those failures could be the drive letter issue. Let's land the fix above and then review the stats.. if there are still failures and that wasn't a one-off, I'll open a change to add verbose output and we can go from there.

(I'll ping back here a little after that fix lands)

[DanTup]

A fix for the second issue (drive casing) has landed. I believe the first issue is the same as https://github.com/dart-lang/sdk/issues/55313 - let me know if we should close this as a dupe, or it's better to keep this open until that's also fixed (I don't know how the flaky test approval works if it might be tied to this?).

[DanTup]

@derekxu16 are you able to review what this dashboard looks like now with some of the fixes above landed?

[derekxu16]

Clicking on the numbers in the RTE and TO columns shows me the last time a result in the specified category occurred. The test hasn't timed out on either win-x64 or win-arm64 since May 2, 2024 (https://ci.chromium.org/ui/p/dart/builders/ci.sandbox/pkg-win-release/28313/overview and https://ci.chromium.org/ui/p/dart/builders/ci.sandbox/pkg-win-release-arm64/1453/overview). There hasn't been an RTE on win-x64 since May 30, 2024 (https://ci.chromium.org/ui/p/dart/builders/ci.sandbox/pkg-win-release/28645/overview), but there was an RTE on win-arm64 on August 16, 2024 (https://ci.chromium.org/ui/p/dart/builders/ci.sandbox/pkg-win-release-arm64/2347/overview).

[DanTup]

Thanks - if I'm looking at the right failure, the recent failure is "The pipe is being closed" which I think is the same issue discussed in https://github.com/dart-lang/sdk/issues/55313 (this needs some further input, as I've not been able to come up with any way to handle that error without the VM hanging at the end - I've added example code in that issue).

[DanTup]

I think the original issue here may be resolved by the fix for https://github.com/dart-lang/sdk/issues/55313, however looking at https://dart-ci.firebaseapp.com/#showLatestFailures=false&test=pkg/dds/test/dap/integration/debug_exceptions_test, there are some different recent failures:

https://dart-ci.appspot.com/log/pkg-linux-release/unittest-asserts-release-linux-x64/28665/pkg/dds/test/dap/integration/debug_exceptions_test

00:10 +6 -1: debug mode parses line/column information from stack traces [E]

  Expected: '/b/s/w/itrrr2vys3/dart-sdk-dap-testDGOUGT/appUWATOH/test_file.dart'
    Actual: <null>
     Which: not an <Instance of 'String'>

  package:matcher/src/expect/expect.dart 149:31                 fail
  package:matcher/src/expect/expect.dart 144:3                  _expect
  package:matcher/src/expect/expect.dart 56:3                   expect
  pkg/dds/test/dap/integration/debug_exceptions_test.dart 91:7  main.<fn>.<fn>
  ===== asynchronous gap ===========================
  package:test_api/src/backend/declarer.dart 229:9              Declarer.test.<fn>.<fn>
  ===== asynchronous gap ===========================
  package:test_api/src/backend/declarer.dart 227:7              Declarer.test.<fn>
  ===== asynchronous gap ===========================
  package:test_api/src/backend/invoker.dart 258:9               Invoker._waitForOutstandingCallbacks.<fn>

00:10 +6 -1: Some tests failed.

I'll try to repro this.