search

dart-lang / sdk

The Dart SDK, including the VM, JS and Wasm compilers, analysis, core libraries, and more.
https://dart.dev
BSD 3-Clause "New" or "Revised" License
10.27k stars 1.58k forks source link

Move benchmarking aspect of vm/dart/transferable_test out into golem benchmark #37520

Open aam opened 5 years ago

aam commented 5 years ago

Having benchmarking done on buildbot is source of flakiness.

alexmarkov commented 5 years ago

Recent flaky failure:

 /=======================================================================\
 | vm/dart/transferable_test broke (Pass -> RuntimeError, expected Pass) |
 \=======================================================================/
 --- Command "vm" (took 06.000616s):
 DART_CONFIGURATION=ProductX64 out/ProductX64/dart --ignore-unrecognized-flags --packages=/b/s/w/ir/.packages /b/s/w/ir/runtime/tests/vm/dart/transferable_test.dart
 exit code:
 255
 stdout:
 --- standard
 19ms since last checkin
 22ms since last checkin
 17ms since last checkin
 582ms since last checkin
 20ms since last checkin
 236ms since last checkin
 737ms for round-trip
 205ms since last checkin
 111ms since last checkin
 325ms for round-trip
 182ms since last checkin
 106ms since last checkin
 15ms since last checkin
 294ms for round-trip
 172ms since last checkin
 115ms since last checkin
 25ms since last checkin
 356ms for round-trip
 178ms since last checkin
 14ms since last checkin
 10ms since last checkin
 320ms for round-trip
 152ms since last checkin
 --- transferable
 44ms since last checkin
 21ms since last checkin
 6ms since last checkin
 8ms since last checkin
 10ms since last checkin
 6ms since last checkin
 13ms since last checkin
 8ms since last checkin
 18ms since last checkin
 25ms since last checkin
 33ms since last checkin
 484ms since last checkin
 8ms since last checkin
 8ms since last checkin
 15ms since last checkin
 11ms since last checkin
 10ms since last checkin
 6ms since last checkin
 311ms since last checkin
 12ms since last checkin
 563ms for round-trip
 441ms since last checkin
 8ms since last checkin
 99ms since last checkin
 26ms since last checkin
 607ms for round-trip
 258ms since last checkin
 260ms for round-trip
 91ms since last checkin
 13ms since last checkin
 168ms for round-trip
 76ms since last checkin
 7ms since last checkin
 15ms since last checkin
 17ms since last checkin
 343ms since last checkin
 575ms for round-trip
 15ms since last checkin
 42ms since last checkin
 12ms since last checkin
 standard(2685 ms)/transferable(3035 ms): 0.8846787479406919x
 stderr:
 Unhandled exception:
 Expect.isTrue(false) fails.
 #0      Expect._fail (package:expect/expect.dart:665:5)
 #1      Expect.isTrue (package:expect/expect.dart:135:5)
 #2      main (file:///b/s/w/ir/runtime/tests/vm/dart/transferable_test.dart:37:10)
 #3      _RootZone.runUnary (dart:async/zone.dart:1379:54)
 #4      _FutureListener.handleValue (dart:async/future_impl.dart:137:18)
 #5      Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart:678:45)
 #6      Future._propagateToListeners (dart:async/future_impl.dart:707:32)
 #7      Future._completeWithValue (dart:async/future_impl.dart:522:5)
 #8      _AsyncAwaitCompleter.complete (dart:async-patch/async_patch.dart:30:15)
 #9      _completeOnAsyncReturn (dart:async-patch/async_patch.dart:288:13)
 #10     runBatch (file:///b/s/w/ir/runtime/tests/vm/dart/transferable_test.dart)
 #11     _RootZone.runUnary (dart:async/zone.dart:1379:54)
 #12     _FutureListener.handleValue (dart:async/future_impl.dart:137:18)
 #13     Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart:678:45)
 #14     Future._propagateToListeners (dart:async/future_impl.dart:707:32)
 #15     Future._completeWithValue (dart:async/future_impl.dart:522:5)
 #16     Future._asyncComplete.<anonymous closure> (dart:async/future_impl.dart:552:7)
 #17     _microtaskLoop (dart:async/schedule_microtask.dart:41:21)
 #18     _startMicrotaskLoop (dart:async/schedule_microtask.dart:50:5)
 #19     _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:116:13)
 #20     _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:173:5)
 --- Re-run this test:
 python tools/test.py -n dartk-linux-product-x64 vm/dart/transferable_test

Full log.

alexmarkov commented 4 years ago

vm/dart/transferable_test is still failing:

/========================================================================\
 | vm/dart/transferable_test/0 failed again (RuntimeError, expected Pass) |
 \========================================================================/
 --- Command "vm" (took 11.000799s):
 set DART_CONFIGURATION=ReleaseIA32 & set DART_SUPPRESS_WER=1 & set DART_CRASHPAD_HANDLER=C:\b\s\w\ir\out\ReleaseIA32\crashpad_handler.exe & out\ReleaseIA32\dart.exe --enable-isolate-groups --ignore-unrecognized-flags --packages=C:\b\s\w\ir\.packages C:\b\s\w\ir\runtime\tests\vm\dart\transferable_test.dart
 exit code:
 255
 stdout:
 --- standard
 20ms since last checkin
 11ms since last checkin
 47ms since last checkin
 426ms since last checkin
 194ms since last checkin
 22ms since last checkin
 136ms since last checkin
 715ms for round-trip
 332ms since last checkin
 136ms since last checkin
 786ms for round-trip
 393ms since last checkin
 19ms since last checkin
 300ms since last checkin
 864ms for round-trip
 364ms since last checkin
 251ms since last checkin
 898ms for round-trip
 624ms since last checkin
 211ms since last checkin
 866ms for round-trip
 214ms since last checkin
 --- transferable
 24ms since last checkin
 35ms since last checkin
 822ms since last checkin
 181ms since last checkin
 239ms for round-trip
 229ms since last checkin
 14ms since last checkin
 6ms since last checkin
 7ms since last checkin
 15ms since last checkin
 75ms since last checkin
 428ms for round-trip
 173ms since last checkin
 146ms since last checkin
 346ms for round-trip
 312ms since last checkin
 34ms since last checkin
 9ms since last checkin
 61ms since last checkin
 64ms since last checkin
 538ms for round-trip
 149ms since last checkin
 15ms since last checkin
 13ms since last checkin
 1114ms since last checkin
 1310ms for round-trip
 standard(4750 ms)/transferable(4003 ms): 1.186610042468149x
 stderr:
 Unhandled exception:
 Expect.isTrue(false) fails.
 #0      Expect._fail (package:expect/expect.dart:666:5)
 #1      Expect.isTrue (package:expect/expect.dart:135:5)
 #2      main (file:///C:/b/s/w/ir/runtime/tests/vm/dart/transferable_test.dart:40:10)
 <asynchronous suspension>
 #3      _startIsolate.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:301:19)
 #4      _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:168:12)
 --- Re-run this test:
 python tools/test.py -n dartk-win-release-ia32 vm/dart/transferable_test/0

https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket.appspot.com/8889767235324743600/+/steps/test_results/0/logs/ignored_flaky_test_failure_logs/0

alexmarkov commented 4 years ago

Failed once again on dartk-mac-debug-x64 configuration:

/=========================================================================\
| vm/dart/transferable_test/0 broke (Pass -> RuntimeError, expected Pass) |
\=========================================================================/

--- Command "vm" (took 09.000665s):
DART_CONFIGURATION=DebugX64 xcodebuild/DebugX64/dart --enable-isolate-groups --ignore-unrecognized-flags --packages=/b/s/w/ir/.packages /b/s/w/ir/runtime/tests/vm/dart/transferable_test.dart

exit code:
255

stdout:
--- standard
6ms since last checkin
62ms since last checkin
10ms since last checkin
376ms since last checkin
25ms since last checkin
120ms since last checkin
141ms since last checkin
746ms for round-trip
317ms since last checkin
150ms since last checkin
35ms since last checkin
465ms for round-trip
206ms since last checkin
111ms since last checkin
11ms since last checkin
495ms for round-trip
426ms since last checkin
135ms since last checkin
15ms since last checkin
643ms for round-trip
353ms since last checkin
11ms since last checkin
130ms since last checkin
20ms since last checkin
608ms for round-trip
392ms since last checkin
56ms since last checkin
--- transferable
26ms since last checkin
6ms since last checkin
6ms since last checkin
19ms since last checkin
8ms since last checkin
6ms since last checkin
6ms since last checkin
8ms since last checkin
119ms since last checkin
1461ms since last checkin
6ms since last checkin
23ms since last checkin
35ms since last checkin
205ms for round-trip
169ms since last checkin
27ms since last checkin
34ms since last checkin
244ms for round-trip
762ms since last checkin
75ms since last checkin
33ms since last checkin
109ms since last checkin
8ms since last checkin
172ms since last checkin
220ms since last checkin
1367ms for round-trip
71ms since last checkin
38ms since last checkin
65ms since last checkin
213ms for round-trip
72ms since last checkin
85ms since last checkin
16ms since last checkin
33ms since last checkin
249ms for round-trip
27ms since last checkin
86ms since last checkin
standard(3743 ms)/transferable(4367 ms): 0.8571101442637966x

stderr:
Unhandled exception:
Expect.isTrue(false) fails.
#0      Expect._fail (package:expect/expect.dart:677:5)
#1      Expect.isTrue (package:expect/expect.dart:135:5)
#2      main (file:///b/s/w/ir/runtime/tests/vm/dart/transferable_test.dart:40:10)
<asynchronous suspension>
#3      _startIsolate.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:301:19)
#4      _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:168:12)

--- Re-run this test:
python tools/test.py -n dartk-mac-debug-x64 vm/dart/transferable_test/0