[Epic] Flaky tests roundup

MartinZikmund commented 2 years ago

Current behavior

This epic tracks flaky tests so they can be addressed one by one.

When a flaky test is added in this list, it can be set to [Ignore] on the platform or platforms where it tends to fail.

When_Rectangle UI test is very flaky on iOS
BreadcrumbBar leak tests are flaky on iOS
Given_AppWindow.WhenResize is failing in CI on Linux, passing locally
[ ] #17101
[ ] #17102
[ ] When_Image_Is_Monochromatic fails on iOS .NET 9 RC1
[ ] https://github.com/unoplatform/uno/issues/18064
[ ] Some Given_ImageBrush.WhenStretch fail on iOS
[ ] When_Hungarian_Culture_Column_Order, When_Czech_Culture_Column_Order and When_CanadaFrench_Culture_Column_Order fail on net9 wasm. It could be related to a missing/filtered ICU, or a headless browser issue, it works locally.
[ ] MoveItemsBetweenPrimaryAndSecondaryCommands is unstable on android

[ ] When_DownSampling is flaky and disabled in https://github.com/unoplatform/uno/pull/16991

Running test When_DownSampling()
GLIB UNHANDLED EXCEPTIONSystem.ArgumentNullException: Value cannot be null. (Parameter 'bitmap')
   at SkiaSharp.SKImage.FromBitmap(SKBitmap bitmap)
   at Windows.UI.Composition.CompositionSurfaceBrush.UpdatePaint(SKPaint fillPaint, SKRect bounds) in C:\a\1\s\src\Uno.UI.Composition\Composition\CompositionSurfaceBrush.skia.cs:line 91
   at Windows.UI.Composition.CompositionSpriteShape.Paint(PaintingSession& session) in C:\a\1\s\src\Uno.UI.Composition\Composition\CompositionSpriteShape.skia.cs:line 19
   at Windows.UI.Composition.BorderVisual.Paint(PaintingSession& session) in C:\a\1\s\src\Uno.UI.Composition\Composition\BorderVisual.skia.cs:line 127
   at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 190
   at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
   at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
   at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
   at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
   at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
   at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
   at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
   at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
   at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
   at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
   at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
   at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
   at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
   at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
   at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
   at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
   at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
   at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
   at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
   at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
   at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
   at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
   at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
   at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
   at Windows.UI.Composition.Visual.Render(PaintingSession& parentSession) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 199
   at Windows.UI.Composition.Visual.RenderRootVisual(SKSurface surface, Nullable`1 offsetOverride) in C:\a\1\s\src\Uno.UI.Composition\Composition\Visual.skia.cs:line 152
   at Uno.UI.Runtime.Skia.Gtk.SoftwareRenderSurface.OnDrawn(Context cr) in C:\a\1\s\src\Uno.UI.Runtime.Skia.Gtk\Rendering\SoftwareRenderSurface.cs:line 99
   at Gtk.Widget.Drawn_cb(IntPtr inst, IntPtr cr)

[ ] #16167
[ ] https://github.com/unoplatform/uno/issues/18105
[ ] When_UnloadAndReload_Then_ReMaterializeItems
[ ] When_ThemeChange on Wasm
[ ] Android TestProgressRing_InitialState fails
[ ] FlyoutTest_Unloaded
[ ] FlyoutTest_Target is flaky on iOS
[ ] https://github.com/unoplatform/uno/issues/13417
[ ] PasswordBox_RevealInScrollViewer is flaky on iOS
[ ] #15734
[ ] TextBox_VerticalAlignment is flaky on Wasm
[ ] Verify_MenuBarItem_Bounds_Native_Popups is flaky on Android - started being flaky after pull request #13630 (@ramezgerges)
[ ] TextBox_Readonly is flaky on iOS
[ ] When_Drag_Then_DoesNotScroll is not working on Android, the thumb drags when it should not
[ ] CanChangeDisplayModeBeforeLoaded is flaky on Skia
[ ] Holding_Tests.When_InListViewWithoutItemClick fails for missing item on android
[ ] Holding_Tests.When_Basic fails on android
[ ] Holding_Tests.When_InScroll fails on android
[ ] Holding_Tests.When_InListViewWithItemClick fails on android
[ ] TestTextBoxReleaseOutFocused is flaky on WASM
[ ] https://github.com/unoplatform/uno/issues/15829
[ ] When_MultipleItems is not failing sometimes (Timing related ?)
[ ] Flyout_TemplatedParent is flaky on WASM
[ ] https://github.com/unoplatform/uno/issues/15828
[ ] https://github.com/unoplatform/uno/issues/15471
[ ] TestTranslatedTap is flaky on WASM (could be related to https://github.com/unoplatform/uno/issues/13844)
[ ] ValidateOverflowPlacement does not pass on Android
[ ] TestTranslatedTap is failing on android
[ ] DoesNotShowMenuIfSecondaryElementsAreCollapsed does not pass on Android
[ ] When_NestedInSVAndOutOfViewportOnInitialLoad_Then_MaterializedEvenWhenScrollingOnMinorAxis is flaky on WASM
[ ] #13844
[ ] Android When_NestedIRSlowlyChangeViewport_Then_MaterializedNeededItems (Timeout even with retries) (Youssef: Can't repro)
[ ] Android When_Verify_Canvas_With_Outer_Clip (Assert.Fail failed. ImageAssert.HasColorAt @ line 73) (Youssef: Verified - it always fails)
[ ] UITests.Windows_UI_Xaml_Media.GradientBrushTests.GradientsPage crashes the android emulator completely (not just the app)
[ ] Android VerifyNavigationViewItemToolTipPaneDisplayMode (Youssef: Verified - it always crashes locally)
[ ] When_Focused_Element_Transformed is crashing the iOS/Android emulator
[ ] Android VerifyCollectionChangeGetsRespected (Cleanup System.Exception: Event was not raised. in MUXApiTestBase.Cleanup()) (Youssef: test not disabled on master, and always passes for me)
[ ] ListView_ObservableCollection_Unused_Space fails regularly on iOS (Expected: 132.0f But was: 88.0f)
[ ] When_Open_Close_Root_Twice_Keep_State fails on android for missing TreeView items (Youssef: fails locally with null ref)
[ ] When_UsingMultipleLanguages unit test fails when running with net7 or later (DateFormatter is buggy and CultureInfo itself returns wrong values)
[ ] https://github.com/unoplatform/uno/issues/11997
[ ] https://github.com/unoplatform/uno/issues/13257
[ ] When_Grid_RowCollection_Changes fails with invalid child.SizePassedToArrangeOverride
iOS 16+ tests
- [ ] NativeCommandBar_Size fails (The first orientation change fails to be applied)
- [ ] ListView_SelectedItems fails (Timeout ?)
- [ ] When_Flipview_DataTemplateSelector (AssertFailedException: Assert.Fail failed. ImageAssert.HasColorAt @ line 101)
- [ ] When_Add_Remove failure: ColorPicker
- [ ] When_Add_Remove failure: CalendarView
[ ] iOS When_ItemsControl.When_NestedItemsControl_RecycleTemplate (Assert.AreEqual failed. Expected a difference no greater than <1> between expected value <50> and actual value <0>.)
[ ] iOS RepeaterTests.VerifyCorrectionsInNonScrollableDirection
[ ] iOS/Skia VerifyFocusedItemIsRecycledOnCollectionReset (Skia fixed in https://github.com/unoplatform/uno/pull/13074)
[ ] iOS When_WebMessageReceived (Timed out waiting for condition to be met. When_WebMessageReceived():207)
[ ] iOS When_Flipview_Items_Modified (Timed out waiting for equality condition to be met. Expected -1 but last received value was 0.)
[ ] iOS When_Render_Border_GetPixelsAsync (Assert.AreEqual failed. Expected:<<10, 10>>. Actual:<<10.5, 10>>. Invalid Layouted.)
[ ] https://github.com/unoplatform/uno/issues/13095
[ ] [All platforms] ValidateFractionalWidthDoesNotCrash (Assert.Fail failed. Timeout waiting on SpectrumRectangle.Fill to be set.) (Youssef: Test is actually skipped in WinUI. There is not much info on why it's skipped there, but I don't see anything that should trigger property changed for Fill)
[ ] When_StretchAndAlignment on Android 28 (Youssef: Passing for me locally)
[ ] TextBox_Readonly_ShouldNotBringUpKeyboard on Android 30 (The keyboard does not show up on emulators, breaking the test)
[ ] When_PressOnContainerAndReleaseOnNested_Mouse on WASM (fails in CI)
[ ] When_RepeatDuration: It was enabled in https://github.com/unoplatform/uno/pull/13060, but then failures are found and reverted in https://github.com/unoplatform/uno/pull/13089
[ ] https://github.com/unoplatform/uno/issues/13384
[ ] [iOS/Skia/macOS] RepeaterTests.VerifyRepeaterDoesNotLeakItemContainers (Skia started failing in https://github.com/unoplatform/uno/pull/16523)

In Progress

[x] TestTextBoxReleaseOutFocused is flaky on WASM
[x] TestTextBoxTap is very flaky on WASM
[ ] Given_SymbolIcon.Validate_Size fails on all targets (Expected:<20>. Actual:<627.5>) (https://github.com/unoplatform/uno/pull/13069)
[x] Android VerifyDropdownItemTemplateWithNoControl (failed cleanup System.Exception: Event was not raised.) (Youssef: passing for me - https://github.com/unoplatform/uno/pull/13077)
[x] Android VerifyBreadcrumbDefaultAPIValues (failed cleanup System.Exception: Event was not raised.) (Youssef: passing for me - https://github.com/unoplatform/uno/pull/13077)
[x] ValidateOverflowButtonHidesWhenAppropriateWithNoAppBarButtons, ValidateMoreButtonCanShowWithoutSizeChanging, ValidateOverflowButtonHidesWhenAppropriateWithPrimaryAppBarButtons are failing randomly on android. (Youssef: passing for me - https://github.com/unoplatform/uno/pull/13078)
[x] Android/iOS When_EmbeddedResource (resource is not included when build on dotnet build) (Youssef: This appears to only be an issue for Uno.UI solution, ie, not a problem for external customers. Fixing in https://github.com/unoplatform/uno/pull/13079)

[x] DoubleAnimation_Tests on android. These tests fail on android when running one by one, but don't when run with others. (Youssef: Can't repro. Enabling in #13082)

System.Exception : Error while performing WaitFor([unknown], "Failed to find [Completed] on [Text]", null, null, null)
----> System.TimeoutException : Failed to find [Completed] on [Text]

[x] When_Opacity_Paused_With_FillBehaviorHold_Then_Hold - on Android (Youssef: Can't repro. Enabling in #13082)

[x] When_Transforms_Paused_With_FillBehaviorStop_Then_Hold. This test tends to fail when run alone, indicating a race condition, see in the comments for screenshots of the test. (Youssef: Can't repro. Enabling in #13082)

1) Failed : SamplesApp.UITests.Windows_UI_Xaml_Media_Animation.DoubleAnimation_Tests.When_Transforms_Paused_With_FillBehaviorStop_Then_Hold
ImageAssert.AreEqual @ line 85
pixelTolerance: No color tolerance
expected: Initial (When_Transforms_Paused_With_FillBehaviorStop_Then_Hold_Initial.png {Width=1280, Height=800}) in 
{X=615,Y=442,Width=50,Height=50}
actual  : Final (When_Transforms_Paused_With_FillBehaviorStop_Then_Hold_Final.png {Width=1280, Height=800}) in 
{X=615,Y=442,Width=50,Height=50}

[x] iOS/Android Given_ImageBrushStretch (Incorrect RenderTargetBitmap content) (youssef: Issue not related to RenderTargetBitmap. Fixing test in https://github.com/unoplatform/uno/pull/13081)

Done

[x] https://github.com/unoplatform/uno/issues/10164
[x] Android all Basics_AutomatedTransformation (The specified child already has a parent. You must call removeView() on the child's parent first.) (https://github.com/unoplatform/uno/pull/13061)
[x] Android/Skia When_Render_Then_CanRenderOnCanvas (Assert.Fail failed. ImageAssert.HasColorAt @ line 134) (works locally on Skia, fails on Android and being fixed in https://github.com/unoplatform/uno/pull/13070)
[x] Wasm When_Reorder_To_Last_2 (https://github.com/unoplatform/uno/pull/13093)
[x] [All platforms] RepeaterTests.VerifyUIElementsInItemsSource (https://github.com/unoplatform/uno/pull/13074)
[x] [All platforms] RepeaterTests.NestedRepeaterWithDataTemplateScenario (https://github.com/unoplatform/uno/pull/13074)
[x] [All platforms] RepeaterTests.VerifyStoreScenarioCache (https://github.com/unoplatform/uno/pull/13074)
[x] https://github.com/unoplatform/uno/issues/10165 (https://github.com/unoplatform/uno/pull/13056)
[x] https://github.com/unoplatform/uno/pull/13071

Youssef1313 commented 2 years ago

When_Reorder_To_Last_2 on WASM, Android

This test appears to be Wasm only. I'm disabling in https://github.com/unoplatform/uno/pull/9571

jeromelaban commented 1 year ago

Images for When_Transforms_Paused_With_FillBehaviorStop_Then_Hold:

Teardown: When_Transforms_Paused_With_FillBehaviorStop_Then_Hold_When_Transforms_Paused_With_FillBehaviorStop_Then_Hold_-_Tear_down_on_error Initial: Final:

ramezgerges commented 1 year ago

When_UsingMultipleLanguages is acting really off. First off, CultureInfo returns weird patterns e.g. in ft-CA, info.LongTimePattern returns HH 'h' mm instead of HH:mm. A possible hack is to replace the added garbage with a colon e.g. using Regex.Replace(value, " '[^']*' ", ":"), but obviously, that isn't ideal. Another thing is that our DateTimeFormatter implementation diverges a lot from the net7 implementation, so it might need a rewrite or something.

ramezgerges commented 1 year ago

VerifyFocusedItemIsRecycledOnCollectionReset seems to fail on skia due to a problem with dispatching measure/arrange. Items in an ItemsRepeater are only updated in a Measure, and it appears that no Measure is dispatched during the test (even though some paths are dirty for sure). A manual GetRoot(repeater).Measure(Window.Current.Bounds.Size) (where GetRoot just walks up the tree to get the RootVisual) passes the test, but I've failed to find the root of the problem.

Edit: It appears that TestServices.WindowHelper.WaitForIdle().Wait() is causing a deadlock (usually it's called with await instead of Wait()).

Youssef1313 commented 1 year ago

I took a look at When_EmbeddedResource, I think the problem is related to shared projects.

The TargetPath here is SharedProjectEmbeddedFile.txt instead of Embedded/SubFolder1/SubFolder2/SharedProjectEmbeddedFile.txt.

Youssef1313 commented 1 year ago

AssignLinkMetadata is somehow capable of determining the Link correctly. It should have run before AssignTargetPath.

We already have code for that, but it is racing with code in msbuild.

https://github.com/unoplatform/uno/blob/d1d468f240672e6ce8a2d06ad44db60c1b466fd3/src/SourceGenerators/Uno.UI.Tasks/Content/Uno.UI.Tasks.targets#L33-L40

https://github.com/dotnet/msbuild/blob/69a09e4d0ad2a26fab4100e78585c85bc2f9f5a7/src/Tasks/Microsoft.Common.CurrentVersion.targets#L3098-L3100

The evaluation from Uno.UI.Tasks.targets was done first (because it's imported first), then it got cleared by msbuild code from Microsoft.Common.CurrentVersion.targets

unoplatform / uno