allure-framework / allure-csharp

Allure integrations for C# test frameworks
https://allurereport.org/
Apache License 2.0
104 stars 65 forks source link

Failed results generates "passed" statuses #307

Open alfeigo opened 1 year ago

alfeigo commented 1 year ago

Just tried to use https://github.com/allure-framework/allure-csharp/tree/main/Allure.SpecFlowPlugin

Scenario:

language: ru

@123456 @tms:42 @link:http://example.org

Функция: Просто тест что все вместе работает Сценарий: Очень простой сценарий Дано Давайте что-то откроем

Step:

[Given(@"Давайте что-то откроем")]
public async void Givensomething()
{
    Response = await Client.GetAsync(_endpoint);
    Assert.Equal((HttpStatusCode.OK), Response.StatusCode);

}

Looks at the test output: Xunit.Sdk.EqualException Assert.Equal() Failure Expected: OK Actual: Unauthorized at net_api_cukes.UsersResourceSteps.Givensomething() in

And data of resulting json:

{ "uuid": "9c83ed2f1a854e5f879883e797bf23ca", "historyId": "Очень простой сценарий", "fullName": "Очень простой сценарий", "labels": [ { "name": "thread", "value": ".NET Long Running Task" }, { "name": "host", "value": "0000NBB007WQ6M0" }, { "name": "feature", "value": "Просто тест что все вместе работает" }, { "name": "tag", "value": "123456" }, { "name": "tag", "value": "tms:42" }, { "name": "tag", "value": "link:http://example.org" } ], "links": [], "name": "Очень простой сценарий", "status": "passed", "statusDetails": { "known": false, "muted": false, "flaky": false }, "stage": "finished", "steps": [ { "name": "Дано Давайте что-то откроем", "status": "passed", "statusDetails": { "known": false, "muted": false, "flaky": false }, "stage": "finished", "steps": [], "attachments": [], "parameters": [], "start": 1669807961943, "stop": 1669807961984 } ], "attachments": [], "parameters": [], "start": 1669807961939, "stop": 1669807962109 }

undron commented 1 year ago

The issue is in methods: NUnit.Allure.Core.Steps.AllureStepAspect.WrapSync<T> and NUnit.Allure.Core.Steps.AllureStepAspect.WrapAsync<T> @neparij why should it return default(T); in case of exception?

neparij commented 1 year ago

@undron, thanks for attention, but there is something different happens. Example contains Xunit + SpecFlow adapter. I'll check this as soon as possible.

FYI, @undron — WrapSync and WrapAsync handler is throwing exceptions now and should be catched with continuation task (commit)

2.9.4-preview.1 package

undron commented 1 year ago

@undron, thanks for attention, but there is something different happens. Example contains Xunit + SpecFlow adapter. I'll check this as soon as possible. Oh, sorry, just faced with same issue in NUnit adapter, was very frustrating... WrapSync and WrapAsync handler is throwing exceptions now and should be catched with continuation task Thanks! I will check it out!