Specflow 3 on AppCenter Test Cloud doesn't display steps

artur-malendowicz-gain commented 5 years ago

SpecFlow Version:

[x] 3.0
[ ] 2.4
[ ] 2.3
[ ] 2.2
[ ] 2.1
[ ] 2.0
[ ] 1.9

Used Test Runner

[ ] SpecFlow+Runner
[ ] MSTest
[x] NUnit
[ ] Xunit

Version number: 3.0.213

Project Format of the SpecFlow project

[x] Classic project format using packages.config
[ ] Classis project format using <PackageReference> tags
[ ] Sdk-style project format

.feature.cs files are generated using

[ ] SpecFlow.Tools.MsBuild.Generation NuGet package
[x] SpecFlowSingleFileGenerator custom tool

Visual Studio Version

[ ] VS 2019
[x] VS 2017
[ ] VS 2015

`Enable SpecFlowSingleFileGenerator Custom Tool` option in Visual Studio extension settings

[x] Enabled
[ ] Disabled

Are the latest Visual Studio updates installed?

[x] Yes
[ ] No

.NET Framework:

[x] >= .NET 4.5
[ ] before .NET 4.5
[ ] .NET Core 2.0
[ ] .NET Core 2.1
[ ] .NET Core 2.2
[ ] .NET Core 3.0

Test Execution Method:

[ ] Visual Studio Test Explorer
[ ] TFS/VSTS/Azure DevOps – Task – PLEASE SPECIFY THE NAME OF THE TASK
[ ] Command line – PLEASE SPECIFY THE FULL COMMAND LINE

<SpecFlow> Section in app.config or content of specflow.json

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <configSections>
    <section name="specFlow" type="TechTalk.SpecFlow.Configuration.ConfigurationSectionHandler, TechTalk.SpecFlow" />
  </configSections>
  <specFlow>
  </specFlow>
</configuration>

Repro Project

Issue Description

When running tests on AppCenter we have feature and scenario visible but not steps. It was working with Specflow 2.4

Steps to Reproduce

artur-malendowicz-gain commented 5 years ago

david1995 commented 5 years ago

Hi @artur-malendowicz-gain,

when using SpecFlow >= 3, code-behing generation of feature files is not supported using SpecFlowSingleFileGenerator.

To make sure this is not the cause of the problem, I'd ask you to follow our migration guideline for SpecFlow 3: https://specflow.org/2019/updating-to-specflow-3/.

Please try using the SpecFlow.Tools.MsBuild.Generation package for code-behind generation. And make sure to disable the Enable SpecFlowSingleFileGenerator custom tool setting in Visual Studio. Because when you add or edit a feature file to the project and the aforementioned setting is enabled, the file's custom tool will be automatically set resulting in a generation failure.

I'd kindly ask you to retry the test run after following the steps and report back if the problem still persists.

artur-malendowicz-gain commented 5 years ago

@david1995 I have already read that topic and followed it, however regarding using the tool - we are working on macOS and Visual Studio for Mac so there is no SpecFlowSingleFileGenerator custom tool setting. We are running that on AppCenter mainly.

david1995 commented 5 years ago

Ok, then it might be a different issue. I'd still suggest switching to MSBuild generation since this is the easiest and most reliable way to get consistent support of code-behind generation across Windows, OSX and Linux.

After talking to @SabotageAndi about the issue, I may have got a better hint. We changed the output to using NUnit's integrated output system instead of Console.WriteLine. Basically, we are now using the output the way NUnit designed it to work. I assume your issue is related to this change.

artur-malendowicz-gain commented 5 years ago

@david1995 I have messaged AppCenter support and waiting for an answer, however maybe you could verify, on some sample that you have same issue?

Stephen-Mc commented 5 years ago

We are unable to investigate this on our end due to the overheads that would be necessary. SpecFlow is open source and free, so there are limits to how much time we can invest in tackling issues, especially given that we are a very small team. In this case, it looks like the issue is with App Center, so we would rather focus our attention on those things we are able to fix and improve ourselves.

If you are really desperate, you have the option of purchasing professional support by the hour; but given that the issue seems to be elsewhere, I doubt that would be worthwhile.

We are happy to answer any questions that the App Center team may have - you might want to direct them to this issue in case they have questions. If it turns out there are issues on SpecFlow's side, they can also post their findings here.

SabotageAndi commented 3 years ago

Closed because it's an issue in AppCenter. See the linked issue.

github-actions[bot] commented 3 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

SpecFlowOSS / SpecFlow