Open ttranbunnings opened 2 years ago
Please provide a project where we can reproduce this issue. Thanks.
I am running into the same problem - when I have multiple scenarios in a single feature.
As a quick workaround, I used NUnit runner that runs the same tests just fine. Not a perfect solution, but it works.
And a side note - it would still fail if code was built in Debug mode for me. Running tests sequentially was not an option, so I did not check if this fixed the issue.
same problem here.
SpecFlow: 3.9.58 SpecFlow.XUnit: 3.9.58 XUnit: 2.4.1
on Visual Studio 2022 and azure devops build
I still need a project to reproduce the issue to be able to do anything.
Hello,
i'm using a client projet, so i can't share the projet. The project uses Azure and database objects which needs to be disposed. I will try to reproduce the problem.
Regards,
VGib
same problem here.
SpecFlow: 3.9.58 SpecFlow.XUnit: 3.9.58 XUnit: 2.4.1
on Visual Studio 2022 and azure devops build
Hello @VGib, I'm getting the same issues when trying to run feature files with multiple scenarios, I'm using the same stack. Did you manage to find a workaround or fix?
Thanks, Gabriel
Same issue here:
SpecFlow: 3.9.69 SpecFlow.XUnit: 3.9.69 SpecFlow.LivingDoc: 3.9.57 XUnit: 2.4.1 AdCodicem.SpecFlow.MicrosoftDependencyInjection: 1.1.8
Stacktrace:
System.AggregateException : One or more errors occurred. (Invalid argument) (Object reference not set to an instance of an object.)
---- System.IO.IOException : Invalid argument
---- System.NullReferenceException : Object reference not set to an instance of an object.
----- Inner Stack Trace #1 (System.IO.IOException) -----
at System.IO.Enumeration.FileSystemEnumerator`1.FindNextEntry(Byte* entryBufferPtr, Int32 bufferLength)
at System.IO.Enumeration.FileSystemEnumerator`1.MoveNext()
at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
at System.IO.FileSystemWatcher.RunningInstance..ctor(FileSystemWatcher watcher, SafeFileHandle inotifyHandle, String directoryPath, Boolean includeSubdirectories, NotifyFilters notifyFilters, CancellationToken cancellationToken)
at System.IO.FileSystemWatcher.StartRaisingEvents()
at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value)
at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter)
at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__1_0()
at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func`1 changeTokenProducer, Action changeTokenConsumer)
at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(FileConfigurationSource source)
at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder)
at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
at MyTests.DependencyContainerInitializer.CreateConfiguration()
at MyTests.DependencyContainerInitializer.ConfigureServices(IServiceCollection services)
at AdCodicem.SpecFlow.MicrosoftDependencyInjection.DependencyInjectionPlugin.ConfigureServices(IObjectContainer delegateContainer)
at AdCodicem.SpecFlow.MicrosoftDependencyInjection.DependencyInjectionPlugin.<Initialize>b__3_1(Object _, CustomizeScenarioDependenciesEventArgs args)
at TechTalk.SpecFlow.Plugins.RuntimePluginEvents.RaiseCustomizeScenarioDependencies(ObjectContainer scenarioContainer)
at TechTalk.SpecFlow.Infrastructure.ContainerBuilder.CreateScenarioContainer(IObjectContainer testThreadContainer, ScenarioInfo scenarioInfo)
at TechTalk.SpecFlow.Infrastructure.ContextManager.InitializeScenarioContext(ScenarioInfo scenarioInfo)
at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnScenarioInitialize(ScenarioInfo scenarioInfo)
at TechTalk.SpecFlow.TestRunner.OnScenarioInitialize(ScenarioInfo scenarioInfo)
at MyTests.Feature1.ScenarioInitialize(ScenarioInfo scenarioInfo)
at MyTests.Feature1.Test1(String url, String[] exampleTags)
----- Inner Stack Trace #2 (System.NullReferenceException) -----
at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnScenarioEnd()
at TechTalk.SpecFlow.TestRunner.OnScenarioEnd()
at MyTests.Feature1.TestTearDown()
at MyTests.Feature1.System.IDisposable.Dispose()
at ReflectionAbstractionExtensions.DisposeTestClass(ITest test, Object testClass, IMessageBus messageBus, ExecutionTimer timer, CancellationTokenSource cancellationTokenSource)
@304NotModified We got rid of this issue by switching to NUnit.
I still need a project to reproduce the issue to be able to do anything.
I guess this is hard to ask. It's too unstable for that.
Isn't the stacktrace clear?
@304NotModified We got rid of this issue by switching to NUnit.
Thanks. That isn't an option for us (policies), but [assembly: Xunit.CollectionBehavior(DisableTestParallelization = true)]
is fine to us.
@304NotModified If you don't care about running tests in parallel that's fine, but we need to run them in parallel so that's why we switched to NUnit.
same issue
SpecFlow Version
3.9.40
Which test runner are you using?
xUnit
Test Runner Version Number
xunit.runner.visualstudio/2.4.3
.NET Implementation
.NET 5.0
Project Format of the SpecFlow project
Sdk-style project format
.feature.cs files are generated using
SpecFlowSingleFileGenerator custom tool
Test Execution Method
Command line – PLEASE SPECIFY THE FULL COMMAND LINE
SpecFlow Section in app.config or content of specflow.json
No response
Issue Description
Tests randomly failed when run in parallel. Works fine when run in Debug or disable parallelb__1() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 216
at Xunit.Sdk.ExceptionAggregator.Run(Action code) in C:\Dev\xunit\xunit\src\xunit.core\Sdk\ExceptionAggregator.cs:line 73
`
[assembly: Xunit.CollectionBehavior(DisableTestParallelization = true)]
System.NullReferenceException Object reference not set to an instance of an object. at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnScenarioEnd() at TechTalk.SpecFlow.TestRunner.OnScenarioEnd() at BayAge.Test.Features.BayEntry.BayQueryFeature.TestTearDown() at BayAge.Test.Features.BayEntry.BayQueryFeature.System.IDisposable.Dispose() at Xunit.Sdk.ExecutionTimer.Aggregate(Action action) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\ExecutionTimer.cs:line 31 at ReflectionAbstractionExtensions.DisposeTestClass(ITest test, Object testClass, IMessageBus messageBus, ExecutionTimer timer, CancellationTokenSource cancellationTokenSource) in C:\Dev\xunit\xunit\src\xunit.execution\Extensions\ReflectionAbstractionExtensions.cs:line 79 at Xunit.Sdk.TestInvoker
1.<>c__DisplayClass47_0.` System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.) (Object reference not set to an instan...
System.AggregateException One or more errors occurred. (Object reference not set to an instance of an object.) (Object reference not set to an instance of an object.) Exception doesn't have a stacktrace
System.NullReferenceException Object reference not set to an instance of an object. at TechTalk.SpecFlow.Infrastructure.ContextManager.InitializeScenarioContext(ScenarioInfo scenarioInfo) at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnScenarioInitialize(ScenarioInfo scenarioInfo) at TechTalk.SpecFlow.TestRunner.OnScenarioInitialize(ScenarioInfo scenarioInfo)`
Steps to Reproduce
Run
dotnet test
in terminal or using Jetbrain Rider run testsLink to Repro Project
No response