search

SpecFlowOSS / SpecFlow

#1 .NET BDD Framework. SpecFlow automates your testing & works with your existing code. Find Bugs before they happen. Behavior Driven Development helps developers, testers, and business representatives to get a better understanding of their collaboration
https://www.specflow.org/
Other
2.24k stars 754 forks source link

Could not generate test run result reports. An error occurred in the generator. #2026

Closed djanderson77 closed 3 years ago

djanderson77 commented 4 years ago

SpecFlow Version:

Used Test Runner

Version number: MSTest: 2.1.2; Specflow+SpecRunner: 3.3.17

Project Format of the SpecFlow project

.feature.cs files are generated using

Visual Studio Version

Enable SpecFlowSingleFileGenerator Custom Tool option in Visual Studio extension settings

Are the latest Visual Studio updates installed?

.NET Framework:

Test Execution Method:

<SpecFlow> Section in app.config or content of specflow.json

N/A

Repro Project

Issue Description

I have built a project to test an in-house mobile app using Appium and just recently upgraded to SpecFlow 3.3.X from 2.3.2. Previously, the Specflow reports generated without issue. It's only after the upgrade did I get this error. I did have to forcefully remove the NuGet projects for 2.3.2 from the solution via the NuGet console before installing 3.3.X. We did previously have a paid Specflow+ license.

---------- Starting test run ---------- SpecFlow+Runner execution started SpecFlow+Runner 3.3.17 in Framework clr40 in x86 mode execution started TestAdapter Location: [redacted] Current UserName: '[myusername]', MachineName: '[mymachinename]' SpecRun: running tests in [redacted] Output folder configured to "[redacted]" (default) Profile: FINSM.UITests Android DEV Log file path: [redacted].log Checking activation The user license is registered for [companyname] and will expire never. Activation check completed Starting test run Discovering target: 10 Discovering tests from assembly: [redacted].dll Discovered tests from assembly: [redacted].dll Scenario: Access the Login Intro Screen in Login-IntroScreen (target: 10) -> Succeeded on thread #0 test run finished publishing test results test results published generating reports Starting external report generation process Could not generate test run result reports. An error occurred in the generator. External program returned exit code '-2' Failed generating reports Result: all tests passed Total: 1 Succeeded: 1 Ignored: 0 Pending: 0 Skipped: 0 Failed: 0

Execution Time: 00:01:07.7466868 Adding attachments to VSTest Log file: [redacted].log SpecFlow+Runner execution finished ========== Test run finished: 1 Tests run in 1.2 min (1 Passed, 0 Failed, 0 Skipped) ==========

Steps to Reproduce

Uninstall v2.3.2 completely Install Specflow 3.3.X (latest version) with all dependencies for MSTest

SabotageAndi commented 4 years ago

Did you adjust your reporting template. We had a small breaking change there: https://specflow.org/2019/updating-to-specflow-3/ - SpecFlow+ Runner Report Templates

djanderson77 commented 4 years ago

I am using the default report. I saw this when updating and checked the template and made sure that line was there. However, this did give me another idea. I am using a custom output name (I'm actually generating two copies of the report). My .srprofile has this section in them:

<Report>
    <Template outputName="MyCustomName.html" />
    <Template outputName="MyCustomName_{now:s}.html"/>
  </Report>

I do this for CI/CD processes. I upload the reports to a blob storage, then automatically send a link to the non-dated one to the project team, which includes non-developers. The dated reports are for purely for archiving.

If I comment out this section, the report generates with no issues using the default report name.

artumi commented 4 years ago

We are experiencing same problem with same error message and error code. When using default reporting everything is fine, but as soon as we try to use CucumberJson template, error occurs. Are there any updates on this issue?

ozslab commented 3 years ago

@djanderson77 , @artumi : We have fixed the issue. We are planning to rollout the fix as part of our next release of SpecFlow+ Runner.

wayne-davidson-tickle commented 3 years ago

Hi @ozslab - Are you able to confirm the fix will be available via the SpecRun.SpecFlow package. Was there any ETA on the fix?

ozslab commented 3 years ago

Hi @wayne-davidson-tickle - I am happy to announce that we have just released the fix as part of our latest SpecFlow+ Runner release: https://www.nuget.org/packages/SpecRun.SpecFlow/3.4.51

wayne-davidson-tickle commented 3 years ago

Thanks ozslab, my issue must be a different one. I'll investigate further on my side. As an FYI, this is what I see when running this in our CI/CD system.

Starting external program: "mono" "/codebuild/output/src568/src/s3/00/src/BigBird.QA.Tests.E2E.UI/bin/Debug/netcoreapp3.1/SpecFlowPlusRunner/net461/SpecFlow.Plus.Runner.Reporting.exe" "/tmp/tmpYArZJk.tmp" in /codebuild/output/src568/src/s3/00/src/BigBird.QA.Tests.E2E.UI/bin/Debug/netcoreapp3.1 892 | 'mono "/codebuild/output/src568/src/s3/00/src/BigBird.QA.Tests.E2E.UI/bin/Debug/netcoreapp3.1/SpecFlowPlusRunner/net461/SpecFlow.Plus.Runner.Reporting.exe" "/tmp/tmpYArZJk.tmp"' took 377.447ms 893 |   894 | Could not generate test run result reports. An error occurred in the generator. 895 | External program returned exit code '254' 896 | Results File: /codebuild/output/src568/src/s3/00/src/BigBird.QA.Tests.E2E.UI/bin/Debug/netcoreapp3.1/Results/_ip-10-152-24-39_2020-11-09_05_47_40.trx 897

There's no issue if I don't use custom reports.

github-actions[bot] commented 3 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.