`dotnet test` not working on Linux container, but is on Windows: "An exception occurred while invoking executor 'executor://specrun/executorV3.2.31': No such file or directory"

echarrod commented 4 years ago

SpecFlow Version:

[X] 3.1
[ ] 3.0
[ ] 2.4
[ ] 2.3
[ ] 2.2
[ ] 2.1
[ ] 2.0
[ ] 1.9

Used Test Runner

[ ] SpecFlow+Runner
[X] MSTest
[ ] NUnit
[ ] Xunit

Version number: dotnet version: 3.1.202

Project Format of the SpecFlow project

[ ] Classic project format using packages.config
[X] Classic project format using <PackageReference> tags
[ ] Sdk-style project format

.feature.cs files are generated using

[X] SpecFlow.Tools.MsBuild.Generation NuGet package
[ ] SpecFlowSingleFileGenerator custom tool

Visual Studio Version

[X] VS 2019
[ ] VS 2017
[ ] VS 2015

`Enable SpecFlowSingleFileGenerator Custom Tool` option in Visual Studio extension settings

[ ] Enabled
[X] Disabled

Are the latest Visual Studio updates installed?

[X] Yes
[ ] No, I use Visual Studio version <Major>.<Minor>.<Patch>

.NET Framework:

[ ] >= .NET 4.5
[ ] before .NET 4.5
[ ] .NET Core 2.0
[ ] .NET Core 2.1
[ ] .NET Core 2.2
[X] .NET Core 3.0

Test Execution Method:

[ ] Visual Studio Test Explorer
[ ] TFS/VSTS/Azure DevOps – Task – PLEASE SPECIFY THE NAME OF THE TASK
[X] Command line – dotnet test

<SpecFlow> Section in app.config or content of specflow.json

specflow.json:

{
  "bindingCulture": {
    "language": "en-uk"
  },
  "language": {
    "feature": "en-uk"
  }
}

Issue Description

Tests run fine both when I run through Visual Studio and from command line with dotnet test on a Windows machine. When I push to our CI system, which uses a linux based docker image: mcr.microsoft.com/dotnet/core/sdk, which essentially just does this in the container:

export MSBUILDSINGLELOADCONTEXT=1
dotnet test "myspecflowproject.csproj"

the tests look like they pass, but then I get the following error:

test run finished
publishing test results
Not publishing results.
test results published
generating reports
Starting external report generation process
Starting external program: "mono" "/tmp/build/42c050a2/concourse-src/sandbox-integration/MyProject.Integration/bin/Release/netcoreapp3.1/SpecFlowPlusRunner/net45/SpecFlow.Plus.Runner.Reporting.exe" "/tmp/tmpF9efQs.tmp" in 
Log file: file:////tmp/build/42c050a2/concourse-src/sandbox-integration/MyProject.Integration/TestResults/MyProject.Integration_MyProject.Integration_2020-05-20T071347.log
System.ComponentModel.Win32Exception (2): No such file or directory
   at System.Diagnostics.Process.ForkAndExecProcess(String filename, String[] argv, String[] envp, String cwd, Boolean redirectStdin, Boolean redirectStdout, Boolean redirectStderr, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean usesTerminal, Boolean throwOnNoExec)
   at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at TechTalk.SpecRun.Framework.ProcessHelper.RunProcess(ProcessStartInfo processStartInfo)
   at TechTalk.SpecRun.Framework.Services.ExternalProcessTestRunResultReporter.Report(TestRunResult result, TestProfile testProfile)
   at TechTalk.SpecRun.Framework.Execution.ExecutionEngine.ExecuteTestSuite(TestProfile testProfile, TestRunExecutionConfiguration executionConfiguration, IObjectContainer parentObjectContainer, IExecutionContainerBuilder containerBuilder)
   at TechTalk.SpecRun.VisualStudio.TestAdapter.Execution.SingleSourceTestExecutor.ExecuteTestSuiteSync(IExecutionEngine executionEngine, TestProfile profile, TestRunExecutionConfiguration testRunExecutionConfiguration, IObjectContainer parentObjectContainer)
   at TechTalk.SpecRun.VisualStudio.TestAdapter.Execution.SingleSourceTestExecutor.ExecuteTestSuite(IExecutionEngine executionEngine, TestProfile profile, TestRunExecutionConfiguration testRunExecutionConfiguration, IObjectContainer parentObjectContainer)
   at TechTalk.SpecRun.VisualStudio.TestAdapter.Execution.SingleSourceTestExecutor.Run(String source, FilterItem filter)
   at TechTalk.SpecRun.VisualStudio.TestAdapter.Execution.MultiAssemblyTestExecutor.Run()
   at TechTalk.SpecRun.VisualStudio.TestAdapter.Execution.CommonTestExecutionWrapper.Run()
An exception occurred while invoking executor 'executor://specrun/executorV3.2.31': No such file or directory
  √ Should be able to GET multiple Zoom liveness resources from the session [3s 293ms]

Test Run Failed.
Total tests: 31
     Passed: 31
 Total time: 4.6436 Minutes
       Done executing task "Microsoft.TestPlatform.Build.Tasks.VSTestTask" -- FAILED.
   1:7>Done building target "VSTest" in project "MyProject.Integration.csproj" -- FAILED.
   1:7>Done Building Project "/tmp/build/42c050a2/concourse-src/sandbox-integration/MyProject.Integration/MyProject.Integration.csproj" (VSTest target(s)) -- FAILED.

Build FAILED.

       "/tmp/build/42c050a2/concourse-src/sandbox-integration/MyProject.Integration/MyProject.Integration.csproj" (VSTest target) (1:7) ->
       "/tmp/build/42c050a2/concourse-src/sandbox-integration/MyProject.Integration/MyProject.Integration.csproj" (default target) (1:8) ->
       (CoreCompile target) ->

I've tried: 1) Using .NET Core 2.2 instead of 3.1 2) Using Microsoft.NET.Test.SDK 16.5.0 instead of 16.6.1 3) Using the preview versions of the Specflow packages, but since they aren't all available through Visual Studio, it doesn't seem possible to use them at the same time

My csproj:


  <ItemGroup>
    <PackageReference Include="DotNetEnv" Version="1.4.0" />
    <PackageReference Include="Microsoft.CodeAnalysis.FxCopAnalyzers" Version="3.0.0">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.6.1" />
    <PackageReference Include="MSTest.TestAdapter" Version="2.1.1" />
    <PackageReference Include="MSTest.TestFramework" Version="2.1.1" />
    <PackageReference Include="Portable.BouncyCastle" Version="1.8.6.7" />
    <PackageReference Include="SpecFlow" Version="3.1.97" />
    <PackageReference Include="SpecFlow.Tools.MsBuild.Generation" Version="3.1.97" />
    <PackageReference Include="SpecRun.SpecFlow" Version="3.2.31" />
  </ItemGroup>

Steps to Reproduce

To follow

echarrod commented 4 years ago

In typical fashion, I've been stuck on this for days and then right after I post the issue, I manage to fix it.

I removed the package SpecRun.SpecFlow, and added SpecFlow.MsTest, and now it works

👍

SabotageAndi commented 4 years ago

You had using the SpecFlow+ Runner. One dependency to use it on Linux is to have mono installed. This is needed for the report generation. An alternative is to disable the report generation.

As you found a solution an this is the correct behavior (the error could be better), I am closing this issue.

mygithub07 commented 4 years ago

@SabotageAndi How do I disable report generation ? I think I have similar problem and disabling reports might help.

SabotageAndi commented 4 years ago

I would first try to install the Mono dependency to get it work, because you are losing a key feature of the SpecFlow+ Runner.

Disabling the reporting is documented here: https://docs.specflow.org/projects/specflow-runner/en/latest/Profile/Report.html

GoFightNguyen commented 3 years ago

@SabotageAndi I just spent many hours trying to figure out the cause of this same issue. Only real difference is I am using the latest version of the NuGet pkgs, WSL 2, and netcore 3.1

It would be rad if there was some way to have a better error.

SabotageAndi commented 3 years ago

@GoFightNguyen And you fixed it with installing Mono?

GoFightNguyen commented 3 years ago

@SabotageAndi I don't want to have a dependency on mono, so I did not attempt to install it. Maybe it is because my local repo is in WSL 2 and/or because I am using VS Code (so unable to use the Visual Studio SpecFlow extension), but I had issues using mstest with SpecFlow too. At this point, I need to move on with the project so I decided to leave SpecFlow (for now) :(

mygithub07 commented 3 years ago

I also still have this issue. Mono (v 6.12.0.90) is already installed on system , as I am using VS for mac. Here is the full error

2020-10-12T10:00:05.8325780-04:00:Executing test executor at '/usr/bin/dotnet' 'exec “/Users/../../TestApplication.UiTests/bin/Release/netcoreapp3.1/SpecFlowPlusRunner/netcoreapp3.1/TechTalk.SpecRun.Framework.Executor.anycpu.netcoreapp3_1.dll” --remoteRunnerPid 23140 --uniqueId "2020-10-12T100005" --outputFolder “/Users/../../TestApplication.UiTests/obj/TestResults” --callbackPort 32768 --loggerPort 32769' in ‘/Users../../TestApplication.UiTests/bin/Release/netcoreapp3.1’
2020-10-12T10:00:05.8733140-04:00:System.ComponentModel.Win32Exception (2): No such file or directory
   at System.Diagnostics.Process.ForkAndExecProcess(String filename, String[] argv, String[] envp, String cwd, Boolean redirectStdin, Boolean redirectStdout, Boolean redirectStderr, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean usesTerminal, Boolean throwOnNoExec)
   at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()

ZukkyBaig commented 3 years ago

@SabotageAndi <Report disable="true"/> isn't respected. I have added it to the default.srprofile, and when i'm running the tests in a docker container, the logs state:

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.
Starting external program: "mono" "/app/bin/Release/netcoreapp3.1/SpecFlowPlusRunner/net461/SpecFlow.Plus.Runner.Reporting.exe" "/tmp/tmpEPWiA7.tmp" in /app/bin/Release/netcoreapp3.1
System.ComponentModel.Win32Exception (2): No such file or directory
   at System.Diagnostics.Process.ForkAndExecProcess(String filename, String[] argv, String[] envp, String cwd, Boolean redirectStdin, Boolean redirectStdout, Boolean redirectStderr, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean usesTerminal, Boolean throwOnNoExec)
   at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at TechTalk.SpecRun.Framework.ProcessHelper.RunProcess(ProcessStartInfo processStartInfo)
   at TechTalk.SpecRun.Framework.Services.ExternalProcessTestRunResultReporter.Report(TestRunResult result, TestProfile testProfile)
   at TechTalk.SpecRun.Framework.Execution.ExecutionEngine.ExecuteTestSuite(TestProfile testProfile, TestRunExecutionConfiguration executionConfiguration, IObjectContainer parentObjectContainer, IExecutionContainerBuilder containerBuilder)
   at TechTalk.SpecRun.VisualStudio.TestAdapter.Execution.SingleSourceTestExecutor.ExecuteTestSuiteSync(IExecutionEngine executionEngine, TestProfile profile, TestRunExecutionConfiguration testRunExecutionConfiguration, IObjectContainer parentObjectContainer)
   at TechTalk.SpecRun.VisualStudio.TestAdapter.Execution.SingleSourceTestExecutor.ExecuteTestSuite(IExecutionEngine executionEngine, TestProfile profile, TestRunExecutionConfiguration testRunExecutionConfiguration, IObjectContainer parentObjectContainer)
   at TechTalk.SpecRun.VisualStudio.TestAdapter.Execution.SingleSourceTestExecutor.Run(String source, FilterItem filter)
   at TechTalk.SpecRun.VisualStudio.TestAdapter.Execution.MultiAssemblyTestExecutor.Run()
   at TechTalk.SpecRun.VisualStudio.TestAdapter.Execution.CommonTestExecutionWrapper.Run()
An exception occurred while invoking executor 'executor://specrun/executorV3.5.22': No such file or directory

I installed mono using apt-get, like so:

RUN apt-get update RUN apt-get install -y mono-complete

Which works successfully. However, I thought I would point out that the disabling of reports doesn't seem to change anything.

SabotageAndi commented 3 years ago

@ZukkyBaig Linux is case-sensitive for filenames. The Filename has to be Default.srProfile. With your filename we fall back to default values. We never have a look at this file.

adrusz commented 3 years ago

Is a way to split report generation?

"mono" "/app/bin/Release/netcoreapp3.1/SpecFlowPlusRunner/net461/SpecFlow.Plus.Runner.Reporting.exe" "/tmp/tmpEPWiA7.tmp"

To SpecFlow.Plus.Runner.Reporting.exe as a parameter is passed file /tmp/tmpEPWiA7.tmp, so I would like to run dotnet test and as a result, get this file

Is it possible?

SabotageAndi commented 3 years ago

@adrusz No, this is not possible at the moment. What do you want to achieve? And did you had a look at SpecFlow+ LivingDoc?

adrusz commented 3 years ago

My goal is to generate a report from test execution in cucumberJson format (then it will be imported to Jira XRay) For now, I solve this problem with some workaround

FROM mcr.microsoft.com/dotnet/sdk:5.0 as dotnet

COPY [ "./", "/src"]
WORKDIR /src

RUN dotnet restore ./TestSpecFlow.SpecRun.csproj
RUN dotnet build ./TestSpecFlow.SpecRun.csproj
RUN rm /tmp/* -R
RUN dotnet test ./TestSpecFlow.SpecRun.csproj; exit 0

FROM mono:latest as mono
COPY --from=dotnet [ "/src/bin/Debug/net5.0/SpecFlowPlusRunner/net461", "/Reporting"]
COPY --from=dotnet [ "/src/CucumberJson.cshtml", "/Templates/CucumberJson.cshtml"]
COPY --from=dotnet [ "/tmp/tmp*.tmp", "/Reporting"]
WORKDIR /Reporting

RUN mono ./SpecFlow.Plus.Runner.Reporting.exe ./tmp*.tmp

But dotnet test is failed because report generating is enable, otherwise file /tmp/tmp*.tmp will not be generated

SabotageAndi commented 3 years ago

@adrusz What is hindering you to install mono in your docker image for test execution? Then you get the report directly out to the execution and there is no need to split something up.

adrusz commented 3 years ago

@SabotageAndi I tied but then I had some NullReferenceException from SpecFlow.Plus.Runner.Reporting.exe

SabotageAndi commented 3 years ago

@adrusz As we are discussing this in an already closed issue, let's move us this to a new issue or the community forum. We are spamming 4 people with notifications at the moment.

github-actions[bot] commented 3 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

SpecFlowOSS / SpecFlow