search

SpecFlowOSS / SpecFlow

#1 .NET BDD Framework. SpecFlow automates your testing & works with your existing code. Find Bugs before they happen. Behavior Driven Development helps developers, testers, and business representatives to get a better understanding of their collaboration
https://www.specflow.org/
Other
2.22k stars 751 forks source link

specflow.json config file changes behaviour of test post process #2714

Open koreyba opened 11 months ago

koreyba commented 11 months ago

SpecFlow Version

3.x

Which test runner are you using?

MSTest

Test Runner Version Number

3

.NET Implementation

equal or greater .NET Framework 4.6.1

Project Format of the SpecFlow project

Classic project format using <PackageReference> tags

.feature.cs files are generated using

SpecFlow.Tools.MsBuild.Generation NuGet package

Test Execution Method

Visual Studio Test Explorer

SpecFlow Section in app.config or content of specflow.json

No response

Issue Description

Having a specflow project on .NET Framework 4.x And the project contains app.config file When I add (even if it's empty) a specflow.json config file Then behaviour of test failure handling is changed And instead of the correct, specific line of .feauture file failure I get the last line of the .feature file

Error stack trace without specflow.json file:

Test method AdamStudios.RegressionTests.WatermarkFeature.AttachWatermark threw exception: 
System.Exception: Exception of type 'System.Exception' was thrown.
    at ExtensibleWebAutomationCore.Specflow.SpecflowTestBase.GivenIUseAssetsStudio() in C:\vsts\adamui-uiautomation\Source\AutomationFramework\ExtensibleWebAutomationCore\Specflow\SpecflowTestBase.cs:line 56
   at TechTalk.SpecFlow.Bindings.BindingInvoker.InvokeBinding(IBinding binding, IContextManager contextManager, Object[] arguments, ITestTracer testTracer, TimeSpan& duration) in D:\a\1\s\TechTalk.SpecFlow\Bindings\BindingInvoker.cs:line 69
   at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.ExecuteStepMatch(BindingMatch match, Object[] arguments) in D:\a\1\s\TechTalk.SpecFlow\Infrastructure\TestExecutionEngine.cs:line 411
   at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.ExecuteStep(IContextManager contextManager, StepInstance stepInstance) in D:\a\1\s\TechTalk.SpecFlow\Infrastructure\TestExecutionEngine.cs:line 358
   at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.Step(StepDefinitionKeyword stepDefinitionKeyword, String keyword, String text, String multilineTextArg, Table tableArg) in D:\a\1\s\TechTalk.SpecFlow\Infrastructure\TestExecutionEngine.cs:line 475
   at TechTalk.SpecFlow.TestRunner.Given(String text, String multilineTextArg, Table tableArg, String keyword) in D:\a\1\s\TechTalk.SpecFlow\TestRunner.cs:line 75
   at AdamStudios.RegressionTests.WatermarkFeature.FeatureBackground() in C:\vsts\adamui-uiautomation\Source\Tests\AdamStudios.RegressionTests\Watermark.feature:line 6
   at AdamStudios.RegressionTests.WatermarkFeature.AttachWatermark() in C:\vsts\adamui-uiautomation\Source\Tests\AdamStudios.RegressionTests\Watermark.feature:line 5

Please pay attention to the last line: Watermark.feature:line 5

Stack trace with specflow.json file:

Test method AdamStudios.RegressionTests.WatermarkFeature.AttachWatermark threw exception: 
System.Exception: Exception of type 'System.Exception' was thrown.
    at ExtensibleWebAutomationCore.Specflow.SpecflowTestBase.GivenIUseAssetsStudio() in C:\vsts\adamui-uiautomation\Source\AutomationFramework\ExtensibleWebAutomationCore\Specflow\SpecflowTestBase.cs:line 56
   at TechTalk.SpecFlow.Bindings.BindingInvoker.InvokeBinding(IBinding binding, IContextManager contextManager, Object[] arguments, ITestTracer testTracer, TimeSpan& duration) in D:\a\1\s\TechTalk.SpecFlow\Bindings\BindingInvoker.cs:line 69
   at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.ExecuteStepMatch(BindingMatch match, Object[] arguments) in D:\a\1\s\TechTalk.SpecFlow\Infrastructure\TestExecutionEngine.cs:line 411
   at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.ExecuteStep(IContextManager contextManager, StepInstance stepInstance) in D:\a\1\s\TechTalk.SpecFlow\Infrastructure\TestExecutionEngine.cs:line 316
   at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnAfterLastStep() in D:\a\1\s\TechTalk.SpecFlow\Infrastructure\TestExecutionEngine.cs:line 132
   at AdamStudios.RegressionTests.WatermarkFeature.ScenarioCleanup()
   at AdamStudios.RegressionTests.WatermarkFeature.AttachWatermark() in C:\vsts\adamui-uiautomation\Source\Tests\AdamStudios.RegressionTests\Watermark.feature:line 70

Please pay attention to the last line, it was changed: Watermark.feature:line 70

The correct line is 5, not 70.

The sad thing it's working such way by default if the project is on .NET (not .NET framework)

I'd like this to be fixed.

Steps to Reproduce

  1. Create a SpecFlow project on .NET Framework
  2. Make so your first step would fail.
  3. Run the test, get the failure log. If you project is default it will say that it failed on string 5 of the Calculator.feature file
  4. Add specflow.json file to the project, make it to be always copied to output folder.
  5. Run the test again and compare results.

Expected: failure line is not changed for .feature file

Actual: failure line is different

Link to Repro Project

No response