search

postsharp / Metalama

Metalama is a Roslyn-based meta-programming framework. Use this repo to report bugs or ask questions.
164 stars 4 forks source link

Feature: On Metalama.Testing.AspectTesting test case failure, open diff in diff tool #280

Closed niklasstich closed 2 months ago

niklasstich commented 3 months ago

When executing tests in a project using the Metalama.Testing.AspectTesting library, on failure of a test case, the expected and actual results of the test execution should be opened in a diff using either the Visual Studio diff feature, the Rider diff feature (depending on whether the solution is opened in Visual Studio or Rider) or the system diff tool. Having a full diff view of all differences between expected and actual result would make fixing more complicated or elaborate aspect tests after changes to the aspect a lot easier.

One snapshot testing library that already supports these features would be https://github.com/VerifyTests/Verify. Verify supports XUnit, which Metalama.Testing.AspectTesting seemingly currently uses and it could be used to entirely handle snapshot management (see https://github.com/VerifyTests/Verify#snapshot-management). As a side note, Verify can already handle source generators too (https://github.com/VerifyTests/Verify.SourceGenerators).

gfraiteur commented 3 months ago

Thank you! I love the feature. It will make it to 2021.2.

prochan2 commented 2 months ago

This feature has been introduced in Metalama 2024.2.1-preview. Thank you for the suggestion!

gfraiteur commented 2 months ago

@niklasstich Looking forward for your feedback.

niklasstich commented 2 months ago

@niklasstich Looking forward for your feedback.

Finally got to test this feature today. Running in Rider, it opens the Rider diff viewer and shows actual on the left, expected on the right. What really makes this workflow a lot easier is that you can immediately accept actual into expected via the diff viewer (see screenshot)!

Only minor gripe: Seems like the comparison between actual and expected ignores trailing newline in expected, whereas the diff viewer shows it as a diff. Other than that, works exactly as I would have expected. Thanks for implementing this!

image

gfraiteur commented 2 months ago

You're not the only one to say that left and right sides are inverted, but it seems to be a problem of DiffEngine because we are calling their API with the proper parameters (see our code].

Indeed, the actual comparison ignores whitespace and EOL, and it depends according to the test settings. I think it's just going to be a "feature".