Utilizing bUnits parser and comparer

[egil]

Hi @SimonCropp et al,

I have been experimenting with Verify.Bunit and have some ideas for improving it, such that it leverages the pretty printer and differ built into bUnit. This ensures that bUnit's blazor:xxx attributes are removed and ignored, and a diff is performed using AngleSharp.Diffing with the same configuration that bUnits "inline snapshot testing" method MarkupMatchces uses.

My attempt at a VerifyBunit type looks like this, with lots of copy paste from the current implementation:

using AngleSharp;
using AngleSharp.Diffing.Core;
using AngleSharp.Dom;
using AngleSharp.Text;
using Bunit.Diffing;
using Bunit.Rendering;
using Microsoft.AspNetCore.Components;
using System.Runtime.CompilerServices;

namespace VerifyTests;

public static class VerifyBunit
{
    private const string BunitFileExtension = "html";

    public static bool Initialized { get; private set; }

    public static void Initialize(bool verifyMarkupOnly = true)
    {
        if (Initialized)
        {
            throw new("Already Initialized");
        }

        Initialized = true;

        InnerVerifier.ThrowIfVerifyHasBeenRun();

        VerifierSettings.RegisterFileConverter<IRenderedFragment>(verifyMarkupOnly ? ConvertFragmentMarkupOnly : ConvertFragment);
        VerifierSettings.AddExtraSettings(settings => settings.Converters.Add(new RenderedFragmentConverter()));
        VerifierSettings.RegisterStringComparer(BunitFileExtension, BunitMarkupComparer);
    }

    private static Task<CompareResult> BunitMarkupComparer(string received, string verified, IReadOnlyDictionary<string, object> context)
    {
        using var parser = new BunitHtmlParser();
        var receivedNodes = parser.Parse(received);
        var verifiedNodes = parser.Parse(verified);
        var diffs = receivedNodes.CompareTo(verifiedNodes);

        var result = diffs.Count == 0
            ? CompareResult.Equal
              : CompareResult.NotEqual(CreateDiffMessage(received, verified, diffs));

        return Task.FromResult(result);
    }

    private static string CreateDiffMessage(string received, string verified, IReadOnlyList<IDiff> diffs)
    {
        var builder = StringBuilderPool.Obtain();
        builder.AppendLine();
        builder.AppendLine("HTML comparison failed. The following errors were found:");

        for (int i = 0; i < diffs.Count; i++)
        {
            builder.Append($"  {i + 1}: ");
            builder.AppendLine(diffs[i] switch
            {
                NodeDiff diff when diff.Target == DiffTarget.Text && diff.Control.Path.Equals(diff.Test.Path, StringComparison.Ordinal)
                    => $"The text in {diff.Control.Path} is different.",
                NodeDiff diff when diff.Target == DiffTarget.Text
                    => $"The expected {NodeName(diff.Control)} at {diff.Control.Path} and the actual {NodeName(diff.Test)} at {diff.Test.Path} is different.",
                NodeDiff diff when diff.Control.Path.Equals(diff.Test.Path, StringComparison.Ordinal)
                    => $"The {NodeName(diff.Control)}s at {diff.Control.Path} are different.",
                NodeDiff diff => $"The expected {NodeName(diff.Control)} at {diff.Control.Path} and the actual {NodeName(diff.Test)} at {diff.Test.Path} are different.",
                AttrDiff diff when diff.Control.Path.Equals(diff.Test.Path, StringComparison.Ordinal)
                    => $"The values of the attributes at {diff.Control.Path} are different.",
                AttrDiff diff => $"The value of the attribute {diff.Control.Path} and actual attribute {diff.Test.Path} are different.",
                MissingNodeDiff diff => $"The {NodeName(diff.Control)} at {diff.Control.Path} is missing.",
                MissingAttrDiff diff => $"The attribute at {diff.Control.Path} is missing.",
                UnexpectedNodeDiff diff => $"The {NodeName(diff.Test)} at {diff.Test.Path} was not expected.",
                UnexpectedAttrDiff diff => $"The attribute at {diff.Test.Path} was not expected.",
                _ => throw new SwitchExpressionException($"Unknown diff type detected: {diffs[i].GetType()}"),
            });
        }

        builder.AppendLine();
        builder.AppendLine("Actual HTML:");
        builder.AppendLine();
        builder.AppendLine(received);
        builder.AppendLine();
        builder.AppendLine("Expected HTML:");
        builder.AppendLine();
        builder.AppendLine(verified);

        return builder.ToPool();

        static string NodeName(ComparisonSource source) => source.Node.NodeType.ToString().ToLowerInvariant();
    }

    private static ConversionResult ConvertFragmentMarkupOnly(IRenderedFragment fragment, IReadOnlyDictionary<string, object> context)
    {
        var markup = fragment.Nodes.ToHtml(DiffMarkupFormatter.Instance).Trim();
        return new ConversionResult(null, BunitFileExtension, markup);
    }

    private static ConversionResult ConvertFragment(IRenderedFragment fragment, IReadOnlyDictionary<string, object> context)
    {
        var markup = fragment.Nodes.ToHtml(DiffMarkupFormatter.Instance).Trim();
        var allNodes = fragment.Nodes.Select(x => x.GetDescendantsAndSelf().Count()).Sum();
        var info = new ComponentInfo(GetInstance(fragment), allNodes);
        return new ConversionResult(info, BunitFileExtension, markup);
    }

    private static IComponent? GetInstance(IRenderedFragment fragment)
    {
        var type = fragment.GetType();
        if (!type.IsGenericType)
        {
            return null;
        }

        var renderComponentInterface = type
            .GetInterfaces()
            .SingleOrDefault(_ =>
                _.IsGenericType &&
                _.GetGenericTypeDefinition() == typeof(IRenderedComponentBase<>));

        if (renderComponentInterface == null)
        {
            return null;
        }

        var instanceProperty = renderComponentInterface.GetProperty("Instance")!;
        return (IComponent)instanceProperty.GetValue(fragment)!;
    }

    private sealed class ComponentInfo
    {
        public string? Component { get; }

        public IComponent? Instance { get; }

        public int NodeCount { get; }

        public ComponentInfo(IComponent? instance, int nodeCount)
        {
            Component = instance?.GetType().FullName;
            Instance = instance;
            NodeCount = nodeCount;
        }
    }

    private sealed class RenderedFragmentConverter : WriteOnlyJsonConverter<IRenderedFragment>
    {
        public override void Write(VerifyJsonWriter writer, IRenderedFragment fragment)
        {
            writer.WriteStartObject();

            writer.WriteMember(fragment, GetInstance(fragment), "Instance");

            writer.WriteMember(fragment, fragment.Markup, "Markup");

            writer.WriteEndObject();
        }
    }
}

The above code compiles with the following package references:

Verify.Xunit
bunit

A few questions/comments:

1. RenderedFragmentConverter does not seem to be used?
2. It seems redundant to convert fragment.Nodes to markup just to convert them back to nodes again in the comparer. But as far as I can tell there is no way around this with Verify, right?
3. As mentioned, this utilizes the differ built-in to bUnit. That has the advantage of avoiding tests breaking if bUnit internally makes changes, and it handles some of the special attributes bUnit adds to the output.
4. This defaults to using the semantic comparer (based on Anglesharp.Diffing) that bUnit comes with, which has been configured to ignore bUnits attribute in the markup, and it includes a custom node matcher that is needed in some circumstances.
5. Ideally, users should not be asserting against the internal state of a component. Even though the component is a public thing, accessing its parameters and other public properties is not considered good practice in frontend testing, since you are interacting and accessing the component in a way that the end user will not. The end user will look at the markup only. Thus, by default, I suggest only verifying the produced markup.
6. I did attempt to set BunitFileExtension to bunit, such that the compare would only be used with it, avoiding any conflicts with non-IRenderedFragment compares that may be registered. But it does not seem to be possible.

I am not super familiar with Verify so I wanted to dump my suggestion here instead of opening a PR.

[SimonCropp]

RenderedFragmentConverter does not seem to be used?

Used here https://github.com/VerifyTests/Verify.Blazor/blob/main/src/Verify.Bunit/VerifyBunit.cs#L28

It seems redundant to convert fragment.Nodes to markup just to convert them back to nodes again in the comparer. But as far as I can tell there is no way around this with Verify, right?

unfortunately no. since comparers are designed to work on the rendered result. but perf wise it is usually not an issue since the comparer on files if the text diff resolves to notequal

As mentioned, this utilizes the differ built-in to bUnit. That has the advantage of avoiding tests breaking if bUnit internally makes changes, and it handles some of the special attributes bUnit adds to the output.

Yep i like this approach

This defaults to using the semantic comparer (based on Anglesharp.Diffing) that bUnit comes with, which has been configured to ignore bUnits attribute in the markup, and it includes a custom node matcher that is needed in some circumstances.

I like it. was not aware buint had a comparer

Ideally, users should not be asserting against the internal state of a component. Even though the component is a public thing, accessing its parameters and other public properties is not considered good practice in frontend testing, since you are interacting and accessing the component in a way that the end user will not. The end user will look at the markup only. Thus, by default, I suggest only verifying the produced markup.

Not sure i agree with that. component state can be helpful in a snapshot. the rendered result can somewhat obscure the state, and also sometime omit some useful info. But i am happy to have a option to exclude component state .

I did attempt to set BunitFileExtension to bunit, such that the compare would only be used with it, avoiding any conflicts with non-IRenderedFragment compares that may be registered. But it does not seem to be possible.

yeah that would be difficult. Verify is aware of "what extensions are text" and treats them differently. but it should be possible. I will take a look at it

I am not super familiar with Verify so I wanted to dump my suggestion here instead of opening a PR.

Based on the above i think a few smaller PRs would work better. so we can discuss the specifics of each individual change

Alternatively. I can do a PR to the bunit codebase that moves this project into yours, and make u the owner of the Verify.BUnit nuget. Obviously still happy to help u out, but it would mean you can take Verify.BUnit any direction u want.

For context when i created Verify.BUnit, i was primarily working in blazor. Now i am back on full time backend work, so not doing any blazor.

[egil]

RenderedFragmentConverter does not seem to be used?

Used here https://github.com/VerifyTests/Verify.Blazor/blob/main/src/Verify.Bunit/VerifyBunit.cs#L28

Yes, I can see it is being referenced. I just haven't found a way to use verify yet that causes the type to be used (if I set a breakpoint in the RenderedFragmentConverter.Write method). So I am not sure when it would be used.

It seems redundant to convert fragment.Nodes to markup just to convert them back to nodes again in the comparer. But as far as I can tell there is no way around this with Verify, right?

unfortunately no. since comparers are designed to work on the rendered result. but perf wise it is usually not an issue since the comparer on files if the text diff resolves to notequal

Yeah, I get that. It makes sense. The verified content is in text form, so it makes sense that all comparisons are done based on text.

As mentioned, this utilizes the differ built-in to bUnit. That has the advantage of avoiding tests breaking if bUnit internally makes changes, and it handles some of the special attributes bUnit adds to the output.

Yep i like this approach

This defaults to using the semantic comparer (based on Anglesharp.Diffing) that bUnit comes with, which has been configured to ignore bUnits attribute in the markup, and it includes a custom node matcher that is needed in some circumstances.

I like it. was not aware buint had a comparer

Ideally, users should not be asserting against the internal state of a component. Even though the component is a public thing, accessing its parameters and other public properties is not considered good practice in frontend testing, since you are interacting and accessing the component in a way that the end user will not. The end user will look at the markup only. Thus, by default, I suggest only verifying the produced markup.

Not sure i agree with that. component state can be helpful in a snapshot. the rendered result can somewhat obscure the state, and also sometime omit some useful info. But i am happy to have a option to exclude component state .

It is not a hard rule, we do expose the component instance since there are legitimate cases where you have to do that. But in many cases, the components state is an implementation detail, which tests generally should not depend on. The externally observable behavior of a component, the rendered output, or the external collaborators a component interacts with, is what should be asserted against. Not a hard rule, but a good property to aim for when testing.

I did attempt to set BunitFileExtension to bunit, such that the compare would only be used with it, avoiding any conflicts with non-IRenderedFragment compares that may be registered. But it does not seem to be possible.

yeah that would be difficult. Verify is aware of "what extensions are text" and treats them differently. but it should be possible. I will take a look at it

This is a double edge sword. It would be nice to avoid having the rendered fragment specific compare used for other html based content, but it also makes sense that the file extension on the verified/received files is html. So I guess I was looking for a way to ensure that the comparer here was only used to compare HTML produced by a bUnit IRenderedFragment.

I am not super familiar with Verify so I wanted to dump my suggestion here instead of opening a PR.

Based on the above i think a few smaller PRs would work better. so we can discuss the specifics of each individual change

How would you like to break this up?

Besides the CreateDiffMessage, I think for the most part, all of it goes together. I am not certain if there needs to be changes to the RenderedFragmentConverter since I have not seen it being used though.

Alternatively. I can do a PR to the bunit codebase that moves this project into yours, and make u the owner of the Verify.BUnit nuget. Obviously still happy to help u out, but it would mean you can take Verify.BUnit any direction u want.

I think it makes sense that this stays with Verify, as it has the package prefix and heavily depends on Verify as a framework. There is a certain style and expectations related to how a verify project works that verify users probably have come to expect, and I probably don't have the bandwidth to keep up with that.

For context when i created Verify.BUnit, i was primarily working in blazor. Now i am back on full time backend work, so not doing any blazor.

I am also mostly a backender at work too :)