No matching step definition found when feature files and step definition located in different projects when using NUnit console runner

[Pianykh]

SpecFlow Version

3.9.74

Which test runner are you using?

NUnit

Test Runner Version Number

3.15.2

.NET Implementation

.NET 6.0

Project Format of the SpecFlow project

Sdk-style project format

.feature.cs files are generated using

SpecFlow.Tools.MsBuild.Generation NuGet package

Test Execution Method

Command line – nunit3-console.exe "<path to solution>\Features\bin\Debug\net6.0\Features.dll"

SpecFlow Section in app.config or content of specflow.json

{ "ide": { "editor": { "gherkinFormat": { "indentDataTable": false, "indentExamplesTable": true } } }, "bindingCulture": { "language": "en-us" }, "language": { "feature": "en-us" }, "stepAssemblies": [ { "assembly": "NunitSpecflow" } ] }

Issue Description

1. The Nunit Test project (Features) includes a references to the project (NunitSpecflow) that contains the step bindings. There are two feature files - one with step bindings contains inside "Features" project, another with step bindings which contains in "NunitSpecflow" project.
2. Visual Studio 2022 intelligence does not color the specflow Gherkin steps purple. When you right click on a Gherkin step and select go to definition, the file with step definitions will open for both feature files.
3. When i run tests by test explorer inside Visual Studio both tests pass
4. But when i run tests by NUnit console runner with command: nunit3-console.exe "<path to solution>\Features\bin\Debug\net6.0\Features.dll" I see the log:

NUnit Console 3.15.2 (Release) Copyright (c) 2022 Charlie Poole, Rob Prouse ???????, 1 ???????? 2022 ?. 13:49:44

Runtime Environment OS Version: Microsoft Windows NT 6.2.9200.0 Runtime: .NET Framework CLR v4.0.30319.42000

Test Files D:\Work\NunitSpecflow\NunitSpecflow\Features\bin\Debug\net6.0\Features.dll

Given Given in external project -> No matching step definition found for the step. Use the following code to create one: [Given(@"Given in external project")] public void GivenGivenInExternalProject() { _scenarioContext.Pending(); }

When When in external project -> No matching step definition found for the step. Use the following code to create one: [When(@"When in external project")] public void WhenWhenInExternalProject() { _scenarioContext.Pending(); }

Then Then in external project -> No matching step definition found for the step. Use the following code to create one: [Then(@"Then in external project")] public void ThenThenInExternalProject() { _scenarioContext.Pending(); }

Given Given in same project Given in same project -> done: Steps.GivenGivenInSameProject() (0,0s) When When in same project When in same project -> done: Steps.WhenWhenInSameProject() (0,0s) Then Then in same project Then in same project -> done: Steps.ThenThenInSameProject() (0,0s) -> Loading plugin D:\Work\NunitSpecflow\NunitSpecflow\Features\bin\Debug\net6.0\TechTalk.SpecFlow.NUnit.SpecFlowPlugin.dll -> Loading plugin D:\Work\NunitSpecflow\NunitSpecflow\Features\bin\Debug\net6.0\Features.dll -> Using default config

Run Settings DisposeRunners: True InternalTraceLevel: Verbose WorkDirectory: D:\Work\NunitSpecflow\NunitSpecflow\packages\NUnit.ConsoleRunner.3.15.2\tools ImageRuntimeVersion: 4.0.30319 ImageTargetFrameworkName: .NETCoreApp,Version=v6.0 ImageRequiresX86: False ImageRequiresDefaultAppDomainAssemblyResolver: False TargetRuntimeFramework: netcore-6.0 NumberOfTestWorkers: 8

Test Run Summary Overall result: Passed Test Count: 2, Passed: 1, Failed: 0, Warnings: 0, Inconclusive: 1, Skipped: 0 Start time: 2022-09-01 10:49:44Z End time: 2022-09-01 10:49:45Z Duration: 0.853 seconds

Results (nunit3) saved as TestResult.xml

From the log it becomes clear that the error occurs only for the test whose step definitions are in external project.

Steps to Reproduce

1. Clone solution
2. Build
3. Run tests by NUnit Console runner from directory <path to solution>\packages\NUnit.ConsoleRunner.3.15.2\tools by command nunit3-console.exe "<path to solution>\Features\bin\Debug\net6.0\Features.dll"

Link to Repro Project

https://github.com/Pianykh/NunitSpecflow

[SabotageAndi]

Does it work if you run dotnet test?

[Pianykh]

@SabotageAndi Yes, using dotnet test works correctly

[clrudolphi]

As an experiment I converted the Additional Assembly example to Nunit and observed the same behavior. Works in VS but not with the Nunit console runner.

[clrudolphi]

@Pianykh how are you invoking the nunit-console .exe - by setting your default directory to the location of the runner and then invoking the runner with the full path of the test dll? If so, try the reverse. Set your directory to the location of the test project dll, and invoke the nunit-console with its full path. Such as:

cd \\Features\bin\Debug\net6.0\

\packages\NUnit.ConsoleRunner.3.15.2\tools\nunit-console.exe Features.dll When I did this, it worked fine. It would seem that when the default directory is that of the nunit-console.exe, the specflow.json is not found when searching that directory, thus the additional assemblies are not loaded.

[Pianykh]

@clrudolphi Wow. Yes, it really works, I really appreciate your help. This will help me solve my work problems. But if dig deeper, it seems to me that this is not because the specflow.json file was not found, because if you run the tests in the way specified in the issue, after specifying an assembly to a non-existent project in the specflow.json file, then the resulting error will indicate that assembly does not exist.

[clrudolphi]

The reason I had described it that way was because I had debugged the test from the nunit-console runner using the --debug option and discovered that the specflow.json configuration file was not being respected. Not only were additional Assemblies not included in the configuration, but other elements were set to defaults only (I had set a few items to confirm whether the config was being properly read). That led to the realization that the specflow.json file was not being read because it was not in the default directory (the directory of the nunit-console).

Glad to help.

[Pianykh]

Hmm.. thanks for the description. In any case, this is the solution to my problem, for which I thank you again.

[SabotageAndi]

Yeah, the working directory needs to be the directory of the test assembly, that this all is working.