Windows framework_tests_libraries is 5.32% flaky

fluttergithubbot commented 1 year ago

The post-submit test builder Windows framework_tests_libraries had a flaky ratio 5.32% for the past (up to) 100 commits, which is above our 2.00% threshold.

One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20framework_tests_libraries/12716 Commit: https://github.com/flutter/flutter/commit/97e434f86ab7dfdc87fe895eefba0709c64337d6

Flaky builds: https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20framework_tests_libraries/12716 https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20framework_tests_libraries/12703 https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20framework_tests_libraries/12691 https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20framework_tests_libraries/12641 https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20framework_tests_libraries/12636

Recent test runs: https://flutter-dashboard.appspot.com/#/build?taskFilter=Windows%20framework_tests_libraries

Please follow https://github.com/flutter/flutter/wiki/Reducing-Test-Flakiness#fixing-flaky-tests to fix the flakiness and enable the test back after validating the fix (internal dashboard to validate: go/flutter_test_flakiness).

HansMuller commented 1 year ago

Most of the failures are the usual Skia Gold received an unapproved image in post-submit. One test failed with Skia Gold imgtest init failed. An error occurred when initializing golden file test with goldctl.

https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20framework_tests_libraries/12716

══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
 The following SkiaException was thrown while running async test code:
 Skia Gold received an unapproved image in post-submit
 testing. Golden file images in flutter/flutter are triaged
 in pre-submit during code review for the given PR.
 Visit https://flutter-gold.skia.org// to view and approve
 the image(s), or revert the associated change. For more
 information, visit the wiki:
 https://github.com/flutter/flutter/wiki/Writing-a-golden-file-test-for-package:flutter
 Debug information for Gold --------------------------------
 stdout:    GET https://flutter-gold.skia.org/json/v1/groupings: Get
 "https://flutter-gold.skia.org/json/v1/groupings": read tcp 10.128.0.183:58999->35.201.76.220:443:
 wsarecv: An existing connection was forcibly closed by the remote host.. At common.go:40 retry.go:37
 retry.go:24 common.go:34 goldclient.go:435 goldclient.go:412 goldclient.go:320 goldclient.go:280
 cmd_imgtest.go:363 cmd_imgtest.go:294 command.go:860 command.go:974 command.go:902 command.go:895
 main.go:54 proc.go:250 asm_amd64.s:1598
 stderr: Error running command: ''computing grouping for test "cupertino.switch.tap.off.dark":
 retrieving groupings: making request to https://flutter-gold.skia.org/json/v1/groupings: GET
 https://flutter-gold.skia.org/json/v1/groupings: Get
 "https://flutter-gold.skia.org/json/v1/groupings": read tcp 10.128.0.183:58999->35.201.76.220:443:
 wsarecv: An existing connection was forcibly closed by the remote host.. At common.go:40 retry.go:37
 retry.go:24 common.go:34 goldclient.go:435 goldclient.go:412 goldclient.go:320 goldclient.go:280
 cmd_imgtest.go:363 cmd_imgtest.go:294 command.go:860 command.go:974 command.go:902 command.go:895
 main.go:54 proc.go:250 asm_amd64.s:1598''
 result-state.json: No result file found.
 When the exception was thrown, this was the stack:
 #0      SkiaGoldClient.imgtestAdd (package:flutter_goldens_client/skia_client.dart:243:7)
 <asynchronous suspension>
 #1      MatchesGoldenFile.matchAsync.<anonymous closure> (package:flutter_test/src/_matchers_io.dart:117:32)
 <asynchronous suspension>
 <asynchronous suspension>
 (elided one frame from dart:async)
 The exception was caught asynchronously.
 ════════════════════════════════════════════════════════════════════════════════════════════════════
 02:21 +2382 ~4 -1: C:/b/s/w/ir/x/w/flutter/packages/flutter/test/material/bottom_sheet_theme_test.dart: Modal bottom sheets respond to theme changes
 02:21 +2382 ~4 -1: C:/b/s/w/ir/x/w/flutter/packages/flutter/test/cupertino/switch_test.dart: Switch renders correctly in dark mode [E]
   Test failed. See exception logs above.
   The test description was: Switch renders correctly in dark mode

https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20framework_tests_libraries/12703

08:18 +8750 ~17: C:/b/s/w/ir/x/w/flutter/packages/flutter/test/painting/gradient_test.dart: Transforms Gradients - 90 degrees SweepGradient(center: Alignment.center, startAngle: 0.0, endAngle: 6.3, colors: [Color(0xffffffff), Color(0xff000088)], tileMode: TileMode.clamp, transform: GradientRotation(radians: 1.6))
 ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
 The following SkiaException was thrown while running async test code:
 Skia Gold imgtest init failed.
 An error occurred when initializing golden file test with
 goldctl.
 Debug information for Gold --------------------------------
 stdout:    GET https://flutter-gold.skia.org/json/v1/hashes: Get
 "https://flutter-gold.skia.org/json/v1/hashes": read tcp 10.128.5.204:58919->35.201.76.220:443:
 wsarecv: An existing connection was forcibly closed by the remote host.. At common.go:40 retry.go:37
 retry.go:24 common.go:34 resultstate.go:101 goldclient.go:640 goldclient.go:270 cmd_imgtest.go:285
 cmd_imgtest.go:240 command.go:860 command.go:974 command.go:902 command.go:895 main.go:54
 proc.go:250 asm_amd64.s:1598
 stderr: Error running command: ''downloading from Gold: getting known hashes from
 https://flutter-gold.skia.org/json/v1/hashes (with retries): GET
 https://flutter-gold.skia.org/json/v1/hashes: Get "https://flutter-gold.skia.org/json/v1/hashes":
 read tcp 10.128.5.204:58919->35.201.76.220:443: wsarecv: An existing connection was forcibly closed
 by the remote host.. At common.go:40 retry.go:37 retry.go:24 common.go:34 resultstate.go:101
 goldclient.go:640 goldclient.go:270 cmd_imgtest.go:285 cmd_imgtest.go:240 command.go:860
 command.go:974 command.go:902 command.go:895 main.go:54 proc.go:250 asm_amd64.s:1598''
 When the exception was thrown, this was the stack:
 #0      SkiaGoldClient.imgtestInit (package:flutter_goldens_client/skia_client.dart:188:7)
 <asynchronous suspension>
 #1      FlutterPostSubmitFileComparator.compare (package:flutter_goldens/flutter_goldens.dart:248:5)
 <asynchronous suspension>
 #2      MatchesGoldenFile.matchAsync.<anonymous closure> (package:flutter_test/src/_matchers_io.dart:117:32)
 <asynchronous suspension>
 <asynchronous suspension>
 (elided one frame from dart:async)
 The exception was caught asynchronously.
 ════════════════════════════════════════════════════════════════════════════════════════════════════
 08:18 +8750 ~17 -1: C:/b/s/w/ir/x/w/flutter/packages/flutter/test/painting/gradient_test.dart: Transforms Gradients - 90 degrees SweepGradient(center: Alignment.center, startAngle: 0.0, endAngle: 6.3, colors: [Color(0xffffffff), Color(0xff000088)], tileMode: TileMode.clamp, transform: GradientRotation(radians: 1.6)) [E]
   Test failed. See exception logs above.
   The test description was: SweepGradient(center: Alignment.center, startAngle: 0.0, endAngle: 6.3, colors: [Color(0xffffffff), Color(0xff000088)], tileMode: TileMode.clamp, transform: GradientRotation(radians: 1.6))

https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20framework_tests_libraries/12691

08:47 +8600 ~17: C:/b/s/w/ir/x/w/flutter/packages/flutter/test/material/value_indicating_slider_test.dart: Slider value indicator large text scale
 ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
 The following SkiaException was thrown while running async test code:
 Skia Gold received an unapproved image in post-submit
 testing. Golden file images in flutter/flutter are triaged
 in pre-submit during code review for the given PR.
 Visit https://flutter-gold.skia.org// to view and approve
 the image(s), or revert the associated change. For more
 information, visit the wiki:
 https://github.com/flutter/flutter/wiki/Writing-a-golden-file-test-for-package:flutter
 Debug information for Gold --------------------------------
 stdout:    GET https://flutter-gold.skia.org/json/v1/groupings: Get
 "https://flutter-gold.skia.org/json/v1/groupings": read tcp 10.128.7.112:57450->35.201.76.220:443:
 wsarecv: An existing connection was forcibly closed by the remote host.. At common.go:40 retry.go:37
 retry.go:24 common.go:34 goldclient.go:435 goldclient.go:412 goldclient.go:320 goldclient.go:280
 cmd_imgtest.go:363 cmd_imgtest.go:294 command.go:860 command.go:974 command.go:902 command.go:895
 main.go:54 proc.go:250 asm_amd64.s:1598
 stderr: Error running command: ''computing grouping for test
 "material.slider_m3_start_text_scale_4_width_0": retrieving groupings: making request to
 https://flutter-gold.skia.org/json/v1/groupings: GET
 https://flutter-gold.skia.org/json/v1/groupings: Get
 "https://flutter-gold.skia.org/json/v1/groupings": read tcp 10.128.7.112:57450->35.201.76.220:443:
 wsarecv: An existing connection was forcibly closed by the remote host.. At common.go:40 retry.go:37
 retry.go:24 common.go:34 goldclient.go:435 goldclient.go:412 goldclient.go:320 goldclient.go:280
 cmd_imgtest.go:363 cmd_imgtest.go:294 command.go:860 command.go:974 command.go:902 command.go:895
 main.go:54 proc.go:250 asm_amd64.s:1598''
 result-state.json: No result file found.
 When the exception was thrown, this was the stack:
 #0      SkiaGoldClient.imgtestAdd (package:flutter_goldens_client/skia_client.dart:243:7)
 <asynchronous suspension>
 #1      MatchesGoldenFile.matchAsync.<anonymous closure> (package:flutter_test/src/_matchers_io.dart:117:32)
 <asynchronous suspension>
 <asynchronous suspension>
 (elided one frame from dart:async)
 The exception was caught asynchronously.
 ════════════════════════════════════════════════════════════════════════════════════════════════════
 08:47 +8601 ~17: C:/b/s/w/ir/x/w/flutter/packages/flutter/test/material/value_indicating_slider_test.dart: Slider value indicator large text scale
 08:47 +8601 ~17 -1: C:/b/s/w/ir/x/w/flutter/packages/flutter/test/material/value_indicating_slider_test.dart: Slider value indicator large text scale [E]
   Test failed. See exception logs above.
   The test description was: Slider value indicator large text scale

https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20framework_tests_libraries/12641

00:21 +339 ~1 -1: C:/b/s/w/ir/x/w/flutter/packages/flutter/test/cupertino/date_picker_test.dart: Date picker DatePicker golden tests
 ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
 The following SkiaException was thrown while running async test code:
 Skia Gold received an unapproved image in post-submit
 testing. Golden file images in flutter/flutter are triaged
 in pre-submit during code review for the given PR.
 Visit https://flutter-gold.skia.org// to view and approve
 the image(s), or revert the associated change. For more
 information, visit the wiki:
 https://github.com/flutter/flutter/wiki/Writing-a-golden-file-test-for-package:flutter
 Debug information for Gold --------------------------------
 stdout: Given image with hash ee29362616a282fcbe3a929f21386150 for test
 cupertino.date_picker_test.time.initial
 Untriaged or negative image:
 https://flutter-gold.skia.org/detail?grouping=name%3Dcupertino.date_picker_test.time.initial%26source_type%3Dflutter&digest=ee29362616a282fcbe3a929f21386150
 stderr: Test: cupertino.date_picker_test.time.initial FAIL
 result-state.json: No result file found.
 When the exception was thrown, this was the stack:
 #0      SkiaGoldClient.imgtestAdd (package:flutter_goldens_client/skia_client.dart:243:7)
 <asynchronous suspension>
 #1      MatchesGoldenFile.matchAsync.<anonymous closure> (package:flutter_test/src/_matchers_io.dart:117:32)
 <asynchronous suspension>
 <asynchronous suspension>
 (elided one frame from dart:async)
 The exception was caught asynchronously.
 ════════════════════════════════════════════════════════════════════════════════════════════════════
 ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
 The following SkiaException was thrown while running async test code:
 Skia Gold received an unapproved image in post-submit
 testing. Golden file images in flutter/flutter are triaged
 in pre-submit during code review for the given PR.
 Visit https://flutter-gold.skia.org// to view and approve
 the image(s), or revert the associated change. For more
 information, visit the wiki:
 https://github.com/flutter/flutter/wiki/Writing-a-golden-file-test-for-package:flutter
 Debug information for Gold --------------------------------
 stdout: Given image with hash ad3db1143c8ece1f909115e1d90cd1b5 for test
 cupertino.date_picker_test.date.initial
 Untriaged or negative image:
 https://flutter-gold.skia.org/detail?grouping=name%3Dcupertino.date_picker_test.date.initial%26source_type%3Dflutter&digest=ad3db1143c8ece1f909115e1d90cd1b5
 stderr: Test: cupertino.date_picker_test.date.initial FAIL
 result-state.json: No result file found.
 When the exception was thrown, this was the stack:
 #0      SkiaGoldClient.imgtestAdd (package:flutter_goldens_client/skia_client.dart:243:7)
 <asynchronous suspension>
 #1      MatchesGoldenFile.matchAsync.<anonymous closure> (package:flutter_test/src/_matchers_io.dart:117:32)
 <asynchronous suspension>
 <asynchronous suspension>
 (elided one frame from dart:async)
 The exception was caught asynchronously.

https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20framework_tests_libraries/12636

══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
 The following SkiaException was thrown while running async test code:
 Skia Gold received an unapproved image in post-submit
 testing. Golden file images in flutter/flutter are triaged
 in pre-submit during code review for the given PR.
 Visit https://flutter-gold.skia.org// to view and approve
 the image(s), or revert the associated change. For more
 information, visit the wiki:
 https://github.com/flutter/flutter/wiki/Writing-a-golden-file-test-for-package:flutter
 Debug information for Gold --------------------------------
 stdout:    GET https://flutter-gold.skia.org/json/v1/groupings: Get
 "https://flutter-gold.skia.org/json/v1/groupings": read tcp 10.128.0.173:59156->35.201.76.220:443:
 wsarecv: A connection attempt failed because the connected party did not properly respond after a
 period of time, or established connection failed because connected host has failed to respond.. At
 common.go:40 retry.go:37 retry.go:24 common.go:34 goldclient.go:435 goldclient.go:412
 goldclient.go:320 goldclient.go:280 cmd_imgtest.go:363 cmd_imgtest.go:294 command.go:860
 command.go:974 command.go:902 command.go:895 main.go:54 proc.go:250 asm_amd64.s:1598
 stderr: Error running command: ''computing grouping for test
 "material.bottom_navigation_bar.shifting_transition.7": retrieving groupings: making request to
 https://flutter-gold.skia.org/json/v1/groupings: GET
 https://flutter-gold.skia.org/json/v1/groupings: Get
 "https://flutter-gold.skia.org/json/v1/groupings": read tcp 10.128.0.173:59156->35.201.76.220:443:
 wsarecv: A connection attempt failed because the connected party did not properly respond after a
 period of time, or established connection failed because connected host has failed to respond.. At
 common.go:40 retry.go:37 retry.go:24 common.go:34 goldclient.go:435 goldclient.go:412
 goldclient.go:320 goldclient.go:280 cmd_imgtest.go:363 cmd_imgtest.go:294 command.go:860
 command.go:974 command.go:902 command.go:895 main.go:54 proc.go:250 asm_amd64.s:1598''
 result-state.json: No result file found.
 When the exception was thrown, this was the stack:
 #0      SkiaGoldClient.imgtestAdd (package:flutter_goldens_client/skia_client.dart:243:7)
 <asynchronous suspension>
 #1      MatchesGoldenFile.matchAsync.<anonymous closure> (package:flutter_test/src/_matchers_io.dart:117:32)
 <asynchronous suspension>
 <asynchronous suspension>
 (elided one frame from dart:async)
 The exception was caught asynchronously.
 ════════════════════════════════════════════════════════════════════════════════════════════════════
 06:08 +5482 ~5 -1: C:/b/s/w/ir/x/w/flutter/packages/flutter/test/material/search_anchor_test.dart: The search view default size on different platforms
 06:08 +5482 ~5 -1: C:/b/s/w/ir/x/w/flutter/packages/flutter/test/material/bottom_navigation_bar_test.dart: BottomNavigationBar shifting backgroundColor with transition pump 8 [E]
   Test failed. See exception logs above.
   The test description was: pump 8

 06:08 +5483 ~5 -1: C:/b/s/w/ir/x/w/flutter/packages/flutter/test/material/segmented_button_test.dart: SegmentedButton default overlayColor and foregroundColor resolve pressed state

HansMuller commented 1 year ago

@Piinks - does this look like the symptom of an existing SKIA Gold issue? It looks like https://github.com/flutter/flutter/issues/131758#issuecomment-1662584607

Piinks commented 1 year ago

Yes, this is the same issue tracked in https://issues.skia.org/issues/40045532. Closing.

github-actions[bot] commented 1 year ago

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

flutter / flutter