search

SpecFlowOSS / SpecFlow

#1 .NET BDD Framework. SpecFlow automates your testing & works with your existing code. Find Bugs before they happen. Behavior Driven Development helps developers, testers, and business representatives to get a better understanding of their collaboration
https://www.specflow.org/
Other
2.22k stars 752 forks source link

[BeforeTestRun] and [AfterTestRun] blocks runs for every Test cases in single execution cycle #2652

Closed shubautomation closed 1 year ago

shubautomation commented 1 year ago

SpecFlow Version

3.9.74

Which test runner are you using?

NUnit

Test Runner Version Number

3.13.3

.NET Implementation

.NET Core 3.1

Project Format of the SpecFlow project

Classic project format using <PackageReference> tags

.feature.cs files are generated using

SpecFlow.Tools.MsBuild.Generation NuGet package

Test Execution Method

TFS/VSTS/Azure DevOps – Task – PLEASE SPECIFY THE NAME OF THE TASK

SpecFlow Section in app.config or content of specflow.json

PackageReference Include="ExtentReports" Version="5.0.0-alpha6" /> PackageReference Include="NUnit" Version="3.13.3" /> PackageReference Include="NUnit3TestAdapter" Version="4.2.1" /> PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.3.2" /> PackageReference Include="SpecFlow" Version="3.9.74" /> PackageReference Include="SpecFlow.Tools.MsBuild.Generation" Version="3.9.74" /> PackageReference Include="SpecFlow.NUnit" Version="3.9.74" /> PackageReference Include="Selenium.WebDriver" Version="4.5.1" /> PackageReference Include="Selenium.Support" Version="4.5.1" /> PackageReference Include="log4net" Version="2.0.15" /> PackageReference Include="SeleniumExtras.WaitHelpers" Version="1.0.2" />

Issue Description

I am using Specflow + VS 2019 + + Selenium + nUnit on MAC OS. I have encountered an issue where [BeforeTestRun] and [AfterTestRun] blocks runs for every Test cases in single execution cycle. Lets Say I have ran 2 TCs for one cycle in sequential order( Not in parallel) then execution will be [BeforeTestRun] >> TC1 >> [AfterTestRun] and then again it starts with [BeforeTestRun] >> TC2 >> [AfterTestRun]. Ideally [BeforeTestRun] and [AfterTestRun] should gets executed only once per execution cycle.

Note: It is working as expected in Windows OS.

Steps to Reproduce

  1. Create a project of Specflow + nUnit on Mac OS.
  2. Create feature files, hooks, step defination file
  3. Create method Before Test run and After test run in Hooks
  4. Run 2 Tcs at a time(in sequential order from IDE)
  5. Verify Before Test run and After test blocks gets executed for every Test case

I have created the Sample project where issue is reproducible. please find the sample project attached with this comment. GitHubDemoProject.zip

Link to Repro Project

No response

SabotageAndi commented 1 year ago

I wasn't able to reproduce the issue with your sample, but I found other issues.

shubautomation commented 1 year ago

I wasn't able to reproduce the issue with your sample, but I found other issues.

  • Don't save static instances of Feature/ScenarioContext - you will get very quick into trouble. Always get them via Context Injection
  • Console.WriteLine is not catched by NUnit. You have to use TestContext.WriteLine to get output.

Hi @SabotageAndi

Thanks for the reply. that means for you sample project is working fine? Have you tried on MAC OS? Did you change any package version, code etc ?But it is not working for me means [BeforeTestRun] and [AfterTestRun] is getting executed for every TC in single run. Can you suggest what can be done here. Currently struggling to resolve this issue and it is blocker for me. Any other way or piece of code you like to look into it? Anything related to nUnit.AssemblyHooks? Thanks for the suggestion for 2 points. Will rectify it ASAP.

shubautomation commented 1 year ago

Hi @SabotageAndi

Any you please reply? Still I am facing the same issue. I knew it is not reproducible at your end , but I want to know lets say issue exist then flow will be [OneTimeSetUp] >> [BeforeTestRun] >> Test Case 1 >> [AfterTestRun] >> [OneTimeTearDown] or else [BeforeTestRun] >> [OneTimeSetUp] >> Test Case 1 >> [OneTimeTearDown] >> [AfterTestRun]. I just want to elimnate whether issue is with nUnit or Specflow.

shubautomation commented 1 year ago

Hi @gasparnagy

Can you please help me here as @SabotageAndi is on vacation. It is blocker for me.

gasparnagy commented 1 year ago

@shubautomation I can try to look at it tomorrow.

SabotageAndi commented 1 year ago

@shubautomation You are using additional NUnit setup attributes in your SpecFlow project? Get rid of them. They can't be used together with SpecFlow. Move everything to SpecFlow hooks.

You get only in trouble if you mix these two. SpecFlow is integrating completely into the test runners.

shubautomation commented 1 year ago

[BeforeTestRun] >> Test Case 1 >> [AfterTestRun] >> [OneTimeTearDown]

Hi @SabotageAndi

Thanks for reply. As per my understanding I am not using any Nunit setup attributes. It is nUnit.AssemblyHooks.cs file which is default called and it is created once we installed specflow package to our project.

Also, it is working fine in windows OS.

If you can pinpoint any nunit Attributes which I am using, I will be getting rid of them. But as per my understanding I am not using.

shubautomation commented 1 year ago

@shubautomation I can try to look at it tomorrow.

Hi @gasparnagy thanks for reply. Will wait for your update.

gasparnagy commented 1 year ago

@shubautomation I have reviewed the solution and this looks good. I don't know why you got the problems on Mac (I don't have a Mac, so cannot test it unfortunately).

My suggestions:

I hope this helps.

gasparnagy commented 1 year ago

I close this for now since we cannot reproduce it and there is a workaround. If you get any new input or the workaround does not work, please reopen

github-actions[bot] commented 1 year ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.