Hard-coded 'specflow.json' name doesn't allow different configurations for different test assemblies

[lanfeust69]

SpecFlow Version

3.9.40

Which test runner are you using?

NUnit

Test Runner Version Number

3.13.1

.NET Implementation

.NET Core 3.1

Project Format of the SpecFlow project

Sdk-style project format

.feature.cs files are generated using

SpecFlow.Tools.MsBuild.Generation NuGet package

Test Execution Method

Command line nunit3-console.exe SomeAssembly.dll ; nunit3-console.exe SomeOtherAssembly.dll

SpecFlow Section in app.config or content of specflow.json

# that's the problem : I would like different contents !

Issue Description

We have all our test projects share the same ouput directory (to avoid wasting disk space, and also because there are runtime dependencies (plugins) that are not explicit dependencies of all test projects).

This is fine with the app.config approach, where different test projects can have different configurations (typically to specify step assemblies), but doesn't work with a single hard-coded specflow.json file (even if each test project had their own, they would end up overwriting each other during build). Since this is the only option for .NET Core, we have a regression there.

Maybe an option could be to also a read a <name-of-the-test-assembly>.specflow.json file if it exists ?

Steps to Reproduce

Create two test projects that need different specflow configurations and have the same output directory : only one of them can actually be properly configured.

Link to Repro Project

No response

[SabotageAndi]

Why do you put multiple test assemblies in the same folder?

[lanfeust69]

Why do you put multiple test assemblies in the same folder?

As I said :

to avoid wasting disk space, and also because there are runtime dependencies (plugins) that are not explicit dependencies of all test projects.

And anyway that should be the decision of the developer (or team), not forced upon by the tools they use, especially when it previously used to work (so that's a regression).

[SabotageAndi]

If we are using the word regression here, it is that .NET Core doesn't support app.config anymore. This is not a limitation we did. We added the possibility to use specflow.json for configuration with SpecFlow 2.2.0 (https://github.com/SpecFlowOSS/SpecFlow/blob/master/changelog.txt#L381) about 5 years ago. A lot earlier than SpecFlow got .NET Core support (which happens with 3.0). You can use it in .NET Framework and .NET Core.

I see that this feature is missing, but you are also the first one in 5 years that would like to have it.

But here is the nice thing. SpecFlow is open-source and depends on contributors. I am always happy to review Pull Requests and help with implementing feature requests.

In your case, the code you are looking for is here: https://github.com/SpecFlowOSS/SpecFlow/blob/master/TechTalk.SpecFlow/Configuration/SpecFlowJsonLocator.cs All other configuration related code can be found here: https://github.com/SpecFlowOSS/SpecFlow/tree/master/TechTalk.SpecFlow/Configuration Unit tests are here: https://github.com/SpecFlowOSS/SpecFlow/blob/master/Tests/TechTalk.SpecFlow.RuntimeTests/Configuration/JsonConfigTests.cs Integration Tests are here: https://github.com/SpecFlowOSS/SpecFlow/blob/master/Tests/TechTalk.SpecFlow.Specs/Features/Execution/Configuration.feature

If you want to discuss how to implement it, we can do this here or on Discord (https://discord.com/invite/xQMrjDXx7a)

[lanfeust69]

Indeed, I had taken a look at the code (to convince myself that I had not overlooked a way to specify the configuration file).

I will possibly give it a shot if I find a bit of time, but it is usually better to discuss the topic around an open issue than to come up directly with a PR (at least for non-trivial matters).

[lanfeust69]

By the way, .NET Core does somewhat support app.config, but I fully agree that the additional dependency to the corresponding package is something we'd much rather avoid.

[gabidabet]

Hello I have the same probleme, we have multiple specflow projects and each one with its specific configuration :( How did you solve that problem?

[lanfeust69]

Hello @gabidabet,

Not solved. But for now we don't run .NET Core tests, so we can still live with the app.config files. We also need configuration for the live-doc generation, which in practice needs to go in specflow.json, but we only have a couple of projects that use it, so we can do it sequentially, overwriting the specflow.json file between each project. Obviously, this won't scale, though...

[gabidabet]

Hello @gabidabet,

Not solved. But for now we don't run .NET Core tests, so we can still live with the app.config files. We also need configuration for the live-doc generation, which in practice needs to go in specflow.json, but we only have a couple of projects that use it, so we can do it sequentially, overwriting the specflow.json file between each project. Obviously, this won't scale, though...

Thank you for replying.

[MarkKharitonov]

We would like to use the json configuration, even if our projects are .NET Framework. The reason to have separate json files is because even if locally the code is built into separate directories, the CI server builds it into a shared directory, which is then packaged into the deployment archive to be deployed on the various AT environments and ran there.

I am not a SpecFlow user, my interest lies in the deployment process. But it seems to me the code uses Dependency Injection and so the ISpecFlowJsonLocator interface is injected into the ConfigurationLoader constructor - https://github.com/SpecFlowOSS/SpecFlow/blob/ddab6a3e9c347f32b2dfb7fb89539c737c7a2225/TechTalk.SpecFlow/Configuration/ConfigurationLoader.cs#L22-L27

If I am correct, can we hook into the Dependency Injection configuration process and provide our own implementation of that interface?

[SabotageAndi]

@MarkKharitonov yeah, it uses dependency injection, but it will be resolved earlier than you are able to change it via a plugin.

[MarkKharitonov]

@SabotageAndi - so the DI configuration does not have any hooks? Can you point me at this code? I am not a SpecFlowTest user, but I have a skin in the game when it comes to build, packaging and deployment.

[SabotageAndi]

@MarkKharitonov the DI has hooks, but they are all called after the ISpecFlowJsonLocator is already resolved.

[noeltupas]

@SabotageAndi

https://github.com/SpecFlowOSS/SpecFlow/blob/ddab6a3e9c347f32b2dfb7fb89539c737c7a2225/TechTalk.SpecFlow/Configuration/SpecFlowJsonLocator.cs#L10

What are your thoughts on adding a file look up for "{assemblyname}.specflow.json" as a priority over "specflow.json". Possibly an additional option in the VS extension to "Prioritize assembly specific specflow.json configuration". Does the Locator have visibility to the extension input? If not that, what about build actions for file rename and copy to output directory? Could you point to code where build actions are implemented (i.e. where steps classes are generated)?

Thanks~

[SabotageAndi]

This is not the only location where the file is read. Also, the Visual Studio Extensions (<= VS2019, >= VS2022) and Rider is reading the file. The extensions are not connected to SpecFlow in any way.

But this is only relevant when you are in the IDE, not when you are running scenarios. So for this use case, it doesn't need to be adjusted. So, I think an additional if in https://github.com/SpecFlowOSS/SpecFlow/blob/master/TechTalk.SpecFlow/Configuration/SpecFlowJsonLocator.cs#L10 could be enough.

The problem is the order of these if. I can't remember which of these cases are the generation part and runtime parts.

The copying of the specflow.json to the output directory is done here: https://github.com/SpecFlowOSS/SpecFlow/blob/master/TechTalk.SpecFlow/build/SpecFlow.targets#L16

Entry point of code behind file generation is this MSBuild Task: https://github.com/SpecFlowOSS/SpecFlow/blob/master/SpecFlow.Tools.MsBuild.Generation/GenerateFeatureFileCodeBehindTask.cs

[epresi]

I had an idea in the past how to solve this issue, which as far as I remember covered runtime and generation time as well. I'll try to find my branch when I get home after work.

[gasparnagy]

@lanfeust69 Could you please give some details about which configuration settings you would like to specify in the specflow.json so that we can find some other suggestions?

BTW: I think SpecFlow is still able to read the config from the app.config as well, so an easy solution for you would be to use the app.config instead, I think...

[lanfeust69]

@gasparnagy

@lanfeust69 Could you please give some details about which configuration settings you would like to specify in the specflow.json so that we can find some other suggestions?

I have actually moved from the project where specflow is used, but as I hinted in the initial description, the most crucial setting was specifying the assemblies where the steps are defined.

BTW: I think SpecFlow is still able to read the config from the app.config as well, so an easy solution for you would be to use the app.config instead, I think...

Not on .NET Core...

[EvgenyMosk]

Greetings @gasparnagy!

Could you please give some details about which configuration settings you would like to specify in the specflow.json so that we can find some other suggestions?

TL;DR: I have Assembly C referencing Assembly B, and Assembly B referencing Assembly A:

C -> B -> A

---

I have a similar problem. In my repository we have one project Calculator using SpecFlow steps from another NuGet package. To point SF to that assembly we're using specflow.json (number '1' on the screenshot). But then I have another project/assembly SpecFlowTests where I need steps from Calculator (number '2' on the screenshot). Here I have a project reference to Calculator, as well as specflow.json.

And if I build SpecFlowTests, I'll get only 1 specflow.json in the output directory.

One idea which I'm thinking about is to use a single specflow.json having all the settings, and referencing all the assemblies that I need. But it does look like a hack to me. Maybe you could provide some suggestions?

[gasparnagy]

@EvgenyMosk I have moved over to Reqnroll but this behavior (merging multiple config files) is currently not supported in either tools.

[EvgenyMosk]

Thanks a lot for your reply, @gasparnagy!