Test Runs Failing on net5.0

[mikes-gh]

Describe the bug

Firstly this may not be a bug with bUnit. These are some observations and I would like some advice on how to log repro these issues. Since moving our library to net5.0 our test pipeline has started giving seemingly random failures. We almost exclusively do component testing using bUnit. I am really not sure where to start. Could it possibly be a timing bug with bUnit on Linux (the build server)? I am at a loss since it seems different test fail every time so finding a pattern is very hard as you can imagine. The below is just an example although it probably doesn't help much sorry. This test will pass normally and will almost certainly pass if I just re-run the pipeline.

Example: Testing this component:

<MudButton Variant="Variant.Filled" OnClick="()=>visible=true">Open</MudButton>

<MudDialog @bind-IsVisible="visible">
    <DialogContent>
        <MudText>Wabalabadubdub!</MudText>
    </DialogContent>
    <DialogActions>
        <MudButton Color="Color.Primary" OnClick="()=>visible=false">Close</MudButton>
    </DialogActions>
</MudDialog>

@code {
    public static string __description__ = "Click on open will open the inlined dialog";

    bool visible;
}

With this test:

  [TestFixture]
  public class DialogTests
  {
        private Bunit.TestContext ctx;

        [SetUp]
        public void Setup()
        {
            ctx = new Bunit.TestContext();
            ctx.AddTestServices();
        }

        [TearDown]
        public void TearDown() => ctx.Dispose();

           /// <summary>
        /// Click outside the dialog (or any other method) must update the IsVisible parameter two-way binding on close
        /// </summary>
        /// <returns></returns>
        [Test]
        public async Task InlineDialog_Should_UpdateIsVisibleOnClose()
        {
            var comp = ctx.RenderComponent<MudDialogProvider>();
            comp.Markup.Trim().Should().BeEmpty();
            var service = ctx.Services.GetService<IDialogService>() as DialogService;
            service.Should().NotBe(null);
            // displaying the component with the inline dialog only renders the open button
            var comp1 = ctx.RenderComponent<InlineDialogIsVisibleStateTest>();
            // open the dialog
            comp1.Find("button").Click();
            Console.WriteLine("\nOpened dialog: " + comp.Markup);
            comp.Find("div.mud-dialog-container").Should().NotBe(null);
            // close by click outside
            comp.Find("div.mud-overlay").Click();
            comp.Markup.Trim().Should().BeEmpty();
            // open again
            comp1.Find("button").Click();
            comp.Find("div.mud-dialog-container").Should().NotBe(null);
            // close again by click outside
            comp.Find("div.mud-overlay").Click();
            comp.Markup.Trim().Should().BeEmpty();
        }

    }

  public static class TestContextExtensions
    {
        public static void AddTestServices(this Bunit.TestContext ctx)
        {
            ctx.JSInterop.Mode = JSRuntimeMode.Loose;
            ctx.Services.AddSingleton<NavigationManager>(new MockNavigationManager());
            ctx.Services.AddMudServices(options =>
            {
                options.SnackbarConfiguration.ShowTransitionDuration = 0;
                options.SnackbarConfiguration.HideTransitionDuration = 0;
            });
            ctx.Services.AddScoped(sp => new HttpClient());
            ctx.Services.AddOptions();
        }
    }

Results in this output:

  Failed InlineDialog_Should_UpdateIsVisibleOnClose [25 ms]
  Error Message:
   Bunit.ElementNotFoundException : No elements were found that matches the selector 'div.mud-dialog-container'
  Stack Trace:
     at Bunit.RenderedFragmentExtensions.Find(IRenderedFragment renderedFragment, String cssSelector) in /_/src/bunit.web/Extensions/RenderedFragmentExtensions.cs:line 30
   at MudBlazor.UnitTests.DialogTests.InlineDialog_Should_UpdateIsVisibleOnClose() in /home/vsts/work/1/s/src/MudBlazor.UnitTests/Components/DialogTests.cs:line 123
   at NUnit.Framework.Internal.TaskAwaitAdapter.GenericAdapter`1.GetResult()
   at NUnit.Framework.Internal.AsyncToSyncAdapter.Await(Func`1 invoke)
   at NUnit.Framework.Internal.Commands.TestMethodCommand.RunTestMethod(TestExecutionContext context)
   at NUnit.Framework.Internal.Commands.TestMethodCommand.Execute(TestExecutionContext context)
   at NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.<>c__DisplayClass1_0.<Execute>b__0()
NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.RunTestMethodInThreadAbortSafeZone(TestExecutionContext context, Action action)
  Standard Output Messages:

 Opened dialog: <div class="mud-dialog-container mud-dialog-center"><div class="mud-overlay mud-overlay-dialog" style="" blazor:onclick="2" blazor:onclick:stopPropagation><div class="mud-overlay-scrim mud-overlay-dark mud-overlay-dialog"></div>
         <div class="mud-overlay-content"></div></div>
     <div id="_9e7cbd63e65248b7b09ec03ebe80ff58" class="mud-dialog mud-dialog-width-sm"><div class="mud-dialog-title"><h6 class="mud-typography mud-typography-h6 mud-inherit-text"><svg class="mud-icon-root mud-svg-icon mud-inherit-text mud-icon-size-medium mr-3" focusable="false" viewBox="0 0 24 24" aria-hidden="true"><path d="M0 0h24v24H0z" fill="none"/><path d="M3 17.25V21h3.75L17.81 9.94l-3.75-3.75L3 17.25zM20.71 7.04c.39-.39.39-1.02 0-1.41l-2.34-2.34c-.39-.39-1.02-.39-1.41 0l-1.83 1.83 3.75 3.75 1.83-1.83z"/></svg> Edit rating
         </h6></div><div style="outline-style: none;" blazor:onkeydown="3" blazor:onkeyup="4" blazor:onfocus="5" tabindex="-1" blazor:elementReference=""><div style="pointer-events:none; position:fixed;" tabindex="0" blazor:onfocus="6"></div>

     <div style="pointer-events:none; position:fixed;" tabindex="0" blazor:onfocus="7" blazor:elementReference=""></div>

     <div style="pointer-events:none; position:fixed;" tabindex="-1" blazor:elementReference=""></div>

     <div class="mud-dialog-content"><p>Close and re-open me by clicking out of the dialog. </p></div><div class="mud-dialog-actions"></div>

     <div style="pointer-events:none; position:fixed;" tabindex="0" blazor:onfocus="8" blazor:elementReference=""></div>

     <div style="pointer-events:none; position:fixed;" tabindex="0" blazor:onfocus="9"></div></div></div></div>

Expected behavior: Passes (It does usually)

Version info:

• bUnit version: preview 01
• .NET Runtime and Blazor version: 5.0.3
• OS type and version: buiild server Ubuntu 20.04.2

Additional context: Passes nearly all of the time. This is just one of seeming random failures. All tests that have passed in the past many times. One Thing that has chnaged that seems to have triggered these random failures is our library moving to net 5.0

cc @henon

[mikes-gh]

We are using WaitForAssertion in some of our tests.

[egil]

Thanks for reporting this. I am investigating a very specific test case in bUnit's test suite which always fail on linux, but never on windows.

Can you read through #329 and see if there is an overlap in the tests you are seeing fail and the one describe in that.

[mikes-gh]

I already read it. I was thinking that maybe the WaitForAssertion maybe causing subsequent tests to fail. In particular one of the WaitForAssertion's that we use takes a relatively long time 200ms because we have a delay set in the component to make the UI look better. However we seem to differ in that I think my problems started happening after moving to net 5.0 whereas you are seeing this on netcoreapp3.1 only.

[mikes-gh]

Interestingly if you look at the markup from the Console.Writeline you will see that the element bUnit doesn't find is definitely there. I seem to remember an issue like this earlier that preview 01 resolved. edit or it might have been beta 11

[mikes-gh]

It was this issue https://github.com/egil/bUnit/issues/290

[egil]

A few things I would like to have confirmed:

• Verify you are not using a nightly version of preview 01, but using the latest from nuget.org.
• Beta-11 also supports net5. Was this an issue with beta-11 and .NET5?

With this, and the #329 issue, that only seems to affect Linux, it might have something to do with how the default dispatcher works on Linux.

I might have to set up a VS code instance running on Linux to be able to debug this more easily...

[mikes-gh]

• We are running the preview 01 from nuget.org not the nightly
• We had a different but similar issue on beta 11 as per #290. In that case the same test was failing every-time.
• In this case we are having random failures in different tests.
• We did start to use WaitForAssertion fairly recently.
• We don't always test locally unless doing big changes. The tests always run on the build server for every commit/PR.
• We recently moved our contribution branch to net5 so we are having more commits against net5

[egil]

Thanks for clarifying @mikes-gh.

As for the WaitFor methods... you might need to use them when you have async operations that trigger the renders in the component, otherwise you are likely to see this issue that randomly shows up. That is expected, because the test code runs in its own thread, and the renderer runs in another. And if you trigger something from the test thread, e.g. by clicking a button, that causes the renderer to schedule a async render, e.g. due to a Task.Delay call, then you most definently need to use one of the WaitFor methods.

You can read more about it here https://bunit.egilhansen.com/docs/interaction/awaiting-async-state.html and here https://bunit.egilhansen.com/docs/verification/async-assertion.html

[mikes-gh]

@egil

Understood For our docs website we run all the docs through a render in bUnit just to check they have no errors rendering.

I recently added some table examples that use a web service in OnInitialiseAsync to fetch data. I have seen errors in those tests occasionally. Is this OK?

@using System.Net.Http.Json
@using MudBlazor.Examples.Data.Models
@namespace MudBlazor.Docs.Examples
@inject HttpClient httpClient

<MudTable Items="@Elements.Take(4)" Hover="true" Breakpoint="Breakpoint.Sm">
    <HeaderContent>
        <MudTh>Nr</MudTh>
        <MudTh>Sign</MudTh>
        <MudTh>Name</MudTh>
        <MudTh>Position</MudTh>
        <MudTh>Molar mass</MudTh>
    </HeaderContent>
    <RowTemplate>
        <MudTd DataLabel="Nr">@context.Number</MudTd>
        <MudTd DataLabel="Sign">@context.Sign</MudTd>
        <MudTd DataLabel="Name">@context.Name</MudTd>
        <MudTd DataLabel="Position">@context.Position</MudTd>
        <MudTd DataLabel="Molar mass">@context.Molar</MudTd>
    </RowTemplate>
</MudTable>

@code {
    private IEnumerable<Element> Elements = new List<Element>();

    protected override async Task OnInitializedAsync()
    {
        Elements = await httpClient.GetFromJsonAsync<List<Element>>("webapi/periodictable");
    }    
}

  [Test]
        public void TableBasicExample_Test()
        {
            ctx.RenderComponent<TableBasicExample>();
        }

[egil]

If my assumption that await httpClient.GetFromJsonAsync<List<Element>>("webapi/periodictable"); doesn't complete immediately, then you need to do a WaitFor*.

What happens is that you will see two renders, the first where there is no items in the Elements list, and then, async in the renderers thread, when the awaited http call returns.

You can either use WaitForState or WaitForAssertion, depending on what you are trying to do (there is a WaitForElement #252 planned).

E.g.:

[Test]
public void TableBasicExample_Test()
{
    var cut = ctx.RenderComponent<TableBasicExample>();
    cut.WaitForState(() => cut.FindAll("tbody tr").Count > 0);
}

[mikes-gh]

But if I only need to know it renders without error can I just let if fly?

[egil]

But if I only need to know it renders without error can I just let if fly?

Yes, you can, but that would verify that it renders with content, only that is able to render an empty list.

Question: Is that an example of a test that fails?

[mikes-gh]

I did get this exception. Strange thing is it was a real failure due to the casing of the url but it had sat without causing an exception for a long while before. At what point does the test conclude in the situation above?

A total of 1 test files matched the specified pattern.
  Failed TableBasicExample_Test [25 ms]
  Error Message:
   System.Net.Http.HttpRequestException : Response status code does not indicate success: 404 (No matching mock handler for "GET https://localhost/webapi/periodicTable").
  Stack Trace:
     at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at System.Net.Http.Json.HttpClientJsonExtensions.GetFromJsonAsyncCore[T](Task`1 taskResponse, JsonSerializerOptions options, CancellationToken cancellationToken)
   at MudBlazor.Docs.Examples.TableBasicExample.OnInitializedAsync() in /home/vsts/work/1/s/src/MudBlazor.Docs/Pages/Components/Table/Examples/TableBasicExample.razor:line 28
   at Microsoft.AspNetCore.Components.ComponentBase.RunInitAndSetParametersAsync()
   at Microsoft.AspNetCore.Components.RenderTree.Renderer.GetErrorHandledTask(Task taskToHandle)
   at Bunit.Rendering.TestRenderer.AssertNoUnhandledExceptions() in /_/src/bunit.core/Rendering/TestRenderer.cs:line 280
   at Bunit.Rendering.TestRenderer.Render[TResult](RenderFragment renderFragment, Func`2 activator) in /_/src/bunit.core/Rendering/TestRenderer.cs:line 164
   at Bunit.Rendering.TestRenderer.RenderFragment(RenderFragment renderFragment) in /_/src/bunit.core/Rendering/TestRenderer.cs:line 38
   at Bunit.Extensions.TestContextExtensions.RenderInsideRenderTree[TComponent](TestContextBase testContext, RenderFragment renderFragment) in /_/src/bunit.web/Extensions/TestContextExtensions.cs:line 20
   at Bunit.TestContext.Render[TComponent](RenderFragment renderFragment) in /_/src/bunit.web/TestContext.cs:line 61
   at Bunit.TestContext.RenderComponent[TComponent](ComponentParameter[] parameters) in /_/src/bunit.web/TestContext.cs:line 36
   at MudBlazor.UnitTests.Components.ExampleDocsTests.TableBasicExample_Test() in /home/vsts/work/1/s/src/MudBlazor.UnitTests/Generated/ExampleDocsTests.generated.cs:line 1124
  Skipped DatePicker_Render_Performance [< 1 ms]
  Skipped Open_Close_DateRangePicker_10000_Times_CheckPerformance [< 1 ms]
  Skipped RenderDateRangePicker_10000_Times_CheckPerformance [< 1 ms]

[mikes-gh]

Also I just reran that test with a completely incorrect url and it passes.

[mikes-gh]

It seems to me that there is different threading behaviour on the build server. BTW I am using macOS at the moment. I cant get that same test to fail locally.

[egil]

At what point does the test conclude in the situation above? ... Also I just reran that test with a completely incorrect url and it passes.

In this case, it looks like the RenderComponent method throws, as the httpClient.GetFromJsonAsync throws because of a 404 response. So in this case the error happens before the RenderComponent method returns.

That test error has nothing to do with bUnit though. The exception comes from HttpClient, so that is what you should focus on in this particular case (not saying bUnit not at fault other places). That code would fail with the same exception in a Blazor app, unless HttpClient works differently on Unix vs in the Blazor WASM/Blazor Server.

[mikes-gh]

I think I am wasting your time a bit here. I probably need to take stock and wait for another error. At that point I think I will open a new issue. What do you think?

[egil]

I think I am wasting your time a bit here.

Not necessarily. But you are always welcome to create an issue, or if you simply have at test that you do not understand why it fails, a good place is to post a question with the test, component under test, and output in https://github.com/egil/bUnit/discussions.

Shall we close this issue for now?

[mikes-gh]

Yes I will close. When I have another seemingly random failure I will do my best to tie it down. Certainly there is something going on on the build server that is not happening locally. Whether it be through poorly written tests or some complex threading issues I am not sure.

[mikes-gh]

I had another seemingly random failure but it was this. We render all our docs examples to check for errors. We don't test any assertions. We mock api for http content retrieval using a MockHttpHandler, Every now and then I got an exception . I didn't know why. But when I tested the MockHttpHandler directly it failed every time. I had an incorrect content type. Why does the thread not synchronise with the await call and bubble the exception back to the test thread?

 var response = await HttpClient.GetAsync("https://raw.githubusercontent.com/Garderoben/MudBlazor/master/LICENSE");
        LicenseText = await response.Content.ReadAsStringAsync();

This is the example

@using System.Net
@using System.Text
@namespace MudBlazor.Docs.Examples

<MudDialog DisableSidePadding="true">
    <DialogContent>
        <MudContainer Style="max-height: 300px; overflow-y: scroll">
            @if (Loading)
            {
                <MudProgressCircular Indeterminate="true"></MudProgressCircular>
            }
            else
            {
                <MudText Style="white-space: pre-wrap;">@LicenseText</MudText>
            }
        </MudContainer>
    </DialogContent>
    <DialogActions>
        <MudButton Color="Color.Primary" OnClick="Ok">Accept</MudButton>
    </DialogActions>
</MudDialog>

@code {
    [CascadingParameter] MudDialogInstance MudDialog { get; set; }

    [Inject] HttpClient HttpClient { get; set; }

    protected override async Task OnInitializedAsync()
    {
        await base.OnInitializedAsync();
        Loading = true;
        var response = await HttpClient.GetAsync("https://raw.githubusercontent.com/Garderoben/MudBlazor/master/LICENSE");
        LicenseText = await response.Content.ReadAsStringAsync();
        Loading = false;
    }

    private string LicenseText;
    private bool Loading = false;

    private void Ok()
    {
        MudDialog.Close(DialogResult.Ok(true));
    }
}

And the test which is auto generated just makes sure the example renders without error

        [Test]
        public void DialogScrollableExample_Test()
        {
            ctx.RenderComponent<DialogScrollableExample>();
        }

[egil]

Thanks for taking the time to report this. I am aware of the issue, and will be attempting to fix it in the next release.

[mikes-gh]

@egil Is this the same as #319 ?

[egil]

@egil Is this the same as #319 ?

Exactly.

[egil]

@mikes-gh I just pushed a small change to how the WaitFor logic works, to work around potential race conditions/timing issue. If you want, you can try out the nightly that include the change by following this guide: https://github.com/egil/bUnit/discussions/209

[mikes-gh]

@egil I dont use anyWaitFor in my code. Would it still apply?

[egil]

No, that's unlikely.

[mikes-gh]

So will there be. a fix for the situation where an exception is presented in the test thread on a seemingly random basis. Only very occasionaly does the exception make it to the test thread. Or maybe this is expected if I dont use waitfor?

[egil]

Or maybe this is expected if I dont use waitfor?

No, this should ideally show up no matter what in the test output. It is the plan to fix that in #319.