search

SpecFlowOSS / SpecFlow

#1 .NET BDD Framework. SpecFlow automates your testing & works with your existing code. Find Bugs before they happen. Behavior Driven Development helps developers, testers, and business representatives to get a better understanding of their collaboration
https://www.specflow.org/
Other
2.24k stars 754 forks source link

Random NullReferenceException when run Specflow.xUnit tests in parallel #2536

Open ttranbunnings opened 2 years ago

ttranbunnings commented 2 years ago

SpecFlow Version

3.9.40

Which test runner are you using?

xUnit

Test Runner Version Number

xunit.runner.visualstudio/2.4.3

.NET Implementation

.NET 5.0

Project Format of the SpecFlow project

Sdk-style project format

.feature.cs files are generated using

SpecFlowSingleFileGenerator custom tool

Test Execution Method

Command line – PLEASE SPECIFY THE FULL COMMAND LINE

SpecFlow Section in app.config or content of specflow.json

No response

Issue Description

Tests randomly failed when run in parallel. Works fine when run in Debug or disable parallel[assembly: Xunit.CollectionBehavior(DisableTestParallelization = true)] System.NullReferenceException Object reference not set to an instance of an object. at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnScenarioEnd() at TechTalk.SpecFlow.TestRunner.OnScenarioEnd() at BayAge.Test.Features.BayEntry.BayQueryFeature.TestTearDown() at BayAge.Test.Features.BayEntry.BayQueryFeature.System.IDisposable.Dispose() at Xunit.Sdk.ExecutionTimer.Aggregate(Action action) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\ExecutionTimer.cs:line 31 at ReflectionAbstractionExtensions.DisposeTestClass(ITest test, Object testClass, IMessageBus messageBus, ExecutionTimer timer, CancellationTokenSource cancellationTokenSource) in C:\Dev\xunit\xunit\src\xunit.execution\Extensions\ReflectionAbstractionExtensions.cs:line 79 at Xunit.Sdk.TestInvoker1.<>c__DisplayClass47_0.b__1() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 216 at Xunit.Sdk.ExceptionAggregator.Run(Action code) in C:\Dev\xunit\xunit\src\xunit.core\Sdk\ExceptionAggregator.cs:line 73 `

` System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.) (Object reference not set to an instan...

System.AggregateException One or more errors occurred. (Object reference not set to an instance of an object.) (Object reference not set to an instance of an object.) Exception doesn't have a stacktrace

System.NullReferenceException Object reference not set to an instance of an object. at TechTalk.SpecFlow.Infrastructure.ContextManager.InitializeScenarioContext(ScenarioInfo scenarioInfo) at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnScenarioInitialize(ScenarioInfo scenarioInfo) at TechTalk.SpecFlow.TestRunner.OnScenarioInitialize(ScenarioInfo scenarioInfo)`

Steps to Reproduce

Run dotnet test in terminal or using Jetbrain Rider run tests

Link to Repro Project

No response

SabotageAndi commented 2 years ago

Please provide a project where we can reproduce this issue. Thanks.

avisra commented 2 years ago

I am running into the same problem - when I have multiple scenarios in a single feature.

szarykott commented 2 years ago

As a quick workaround, I used NUnit runner that runs the same tests just fine. Not a perfect solution, but it works.

And a side note - it would still fail if code was built in Debug mode for me. Running tests sequentially was not an option, so I did not check if this fixed the issue.

VGib commented 2 years ago

same problem here.

SpecFlow: 3.9.58 SpecFlow.XUnit: 3.9.58 XUnit: 2.4.1

on Visual Studio 2022 and azure devops build

SabotageAndi commented 2 years ago

I still need a project to reproduce the issue to be able to do anything.

VGib commented 2 years ago

Hello,

i'm using a client projet, so i can't share the projet. The project uses Azure and database objects which needs to be disposed. I will try to reproduce the problem.

Regards,

VGib

burciugi commented 2 years ago

same problem here.

SpecFlow: 3.9.58 SpecFlow.XUnit: 3.9.58 XUnit: 2.4.1

on Visual Studio 2022 and azure devops build

Hello @VGib, I'm getting the same issues when trying to run feature files with multiple scenarios, I'm using the same stack. Did you manage to find a workaround or fix?

Thanks, Gabriel

304NotModified commented 2 years ago

Same issue here:

SpecFlow: 3.9.69 SpecFlow.XUnit: 3.9.69 SpecFlow.LivingDoc: 3.9.57 XUnit: 2.4.1 AdCodicem.SpecFlow.MicrosoftDependencyInjection: 1.1.8

Stacktrace: 

System.AggregateException : One or more errors occurred. (Invalid argument) (Object reference not set to an instance of an object.)
---- System.IO.IOException : Invalid argument
---- System.NullReferenceException : Object reference not set to an instance of an object.

----- Inner Stack Trace #1 (System.IO.IOException) -----
   at System.IO.Enumeration.FileSystemEnumerator`1.FindNextEntry(Byte* entryBufferPtr, Int32 bufferLength)
   at System.IO.Enumeration.FileSystemEnumerator`1.MoveNext()
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance..ctor(FileSystemWatcher watcher, SafeFileHandle inotifyHandle, String directoryPath, Boolean includeSubdirectories, NotifyFilters notifyFilters, CancellationToken cancellationToken)
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
   at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value)
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
   at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__1_0()
   at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func`1 changeTokenProducer, Action changeTokenConsumer)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(FileConfigurationSource source)
   at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at MyTests.DependencyContainerInitializer.CreateConfiguration() 
   at MyTests.DependencyContainerInitializer.ConfigureServices(IServiceCollection services)
   at AdCodicem.SpecFlow.MicrosoftDependencyInjection.DependencyInjectionPlugin.ConfigureServices(IObjectContainer delegateContainer)
   at AdCodicem.SpecFlow.MicrosoftDependencyInjection.DependencyInjectionPlugin.<Initialize>b__3_1(Object _, CustomizeScenarioDependenciesEventArgs args)
   at TechTalk.SpecFlow.Plugins.RuntimePluginEvents.RaiseCustomizeScenarioDependencies(ObjectContainer scenarioContainer)
   at TechTalk.SpecFlow.Infrastructure.ContainerBuilder.CreateScenarioContainer(IObjectContainer testThreadContainer, ScenarioInfo scenarioInfo)
   at TechTalk.SpecFlow.Infrastructure.ContextManager.InitializeScenarioContext(ScenarioInfo scenarioInfo)
   at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnScenarioInitialize(ScenarioInfo scenarioInfo)
   at TechTalk.SpecFlow.TestRunner.OnScenarioInitialize(ScenarioInfo scenarioInfo)
   at MyTests.Feature1.ScenarioInitialize(ScenarioInfo scenarioInfo)
   at MyTests.Feature1.Test1(String url, String[] exampleTags)
----- Inner Stack Trace #2 (System.NullReferenceException) -----
   at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnScenarioEnd()
   at TechTalk.SpecFlow.TestRunner.OnScenarioEnd()
   at MyTests.Feature1.TestTearDown()
   at MyTests.Feature1.System.IDisposable.Dispose()
   at ReflectionAbstractionExtensions.DisposeTestClass(ITest test, Object testClass, IMessageBus messageBus, ExecutionTimer timer, CancellationTokenSource cancellationTokenSource)
burciugi commented 2 years ago

@304NotModified We got rid of this issue by switching to NUnit.

304NotModified commented 2 years ago

I still need a project to reproduce the issue to be able to do anything.

I guess this is hard to ask. It's too unstable for that.

Isn't the stacktrace clear?

@304NotModified We got rid of this issue by switching to NUnit.

Thanks. That isn't an option for us (policies), but [assembly: Xunit.CollectionBehavior(DisableTestParallelization = true)] is fine to us.

burciugi commented 2 years ago

@304NotModified If you don't care about running tests in parallel that's fine, but we need to run them in parallel so that's why we switched to NUnit.

dinaRodeny commented 9 months ago

same issue