search

allure-framework / allure-csharp

Allure integrations for C# test frameworks
https://allurereport.org/
Apache License 2.0
104 stars 64 forks source link

The test case failed using FluentAssertions or Playwright are shown as broken instead of failed #520

Open I37937 opened 2 weeks ago

I37937 commented 2 weeks ago

[//]: # ( . Note: for support questions, please create a discussion at https://github.com/orgs/allure-framework/discussions. . This repository's issues are reserved for feature requests and bug reports. . . Make sure you have a clear name for your issue. The name should start with a capital . letter and no dot is required in the end of the sentence. An example of good issue names: . . - The report is broken in IE11 . - Add an ability to disable default plugins . - Support emoji in test descriptions )

I'm submitting a ...

What is the current behavior?

The test case failed using fluentassertions or Playwright are shown as broken instead of failed

If the current behavior is a bug, please provide the steps to reproduce and, if possible, a minimal demo of the problem

Create a simple test using FluentAssertions or Playwright dotnet, when you run the tests the resutl is shown as broken image

What is the expected behavior?

The test should be failed if I have configured the list of exceptions (failExceptions).

What is the motivation / use case for changing the behavior?

Please tell us about your environment:

Other information

I was taking a look to the AllureNUnitHelper and I was able to see that the IsBroken function is waiting elements in result.Assertions list but when you use FluentAssertions or Playwright this list is not update so the tests are always mark as broken. image

gamerka commented 2 weeks ago

Did you check this page? https://allurereport.org/docs/nunit-configuration/#allurefailexceptions

I37937 commented 2 weeks ago

Yes, I do, I have added the exceptions but I still having the same issue image

arman-avetisyan commented 2 weeks ago

Hello, I have the same issue, I am using Xunit, Allure.Net.Commons and Shouldly When I run a test with this simple step, and generate an Allure report the test status displayed as Broken image

Also, I tried to add the Shouldly.ShouldAssertException to the allureConfig.json, the step becomes Failed but the test is still displayed as Broken image

flydyk commented 2 weeks ago

Looks like this code is making a test report broken despite the settings https://github.com/allure-framework/allure-csharp/blob/main/Allure.Xunit/AllureXunitHelper.cs#L39-L42

            var status = failure.ExceptionTypes.Any(
                exceptionType => !exceptionType.StartsWith("Xunit.Sdk.")
            ) ? Status.broken : Status.failed;