Cannot run tests on Linux container (Ubuntu 16.04) Error message "An exception occurred while invoking executor 'executor://specrun/executorV3.0.352': COM interop is not supported on this platform."

atbe-crowe commented 5 years ago

I'm unable to execute my specflow tests on a linux machine with dotnet and am encountering the following error:

An exception occurred while invoking executor 'executor://specrun/executorV3.0.352': COM interop is not supported on this platform

SpecFlow Version:

[X] 3.0
[ ] 2.4
[ ] 2.3
[ ] 2.2
[ ] 2.1
[ ] 2.0
[ ] 1.9

Used Test Runner

[X] SpecFlow+Runner
[ ] MSTest
[ ] NUnit
[ ] Xunit

Version number: 3.0.352

Project Format of the SpecFlow project

[ ] Classic project format using packages.config
[ ] Classic project format using <PackageReference> tags
[ ] Sdk-style project format

.feature.cs files are generated using

[X] SpecFlow.Tools.MsBuild.Generation NuGet package
[ ] SpecFlowSingleFileGenerator custom tool

Visual Studio Version

[X] VS 2019
[ ] VS 2017
[ ] VS 2015

`Enable SpecFlowSingleFileGenerator Custom Tool` option in Visual Studio extension settings

[ ] Enabled
[X] Disabled

Are the latest Visual Studio updates installed?

[X] Yes
[ ] No, I use Visual Studio version <Major>.<Minor>.<Patch>

.NET Framework:

[ ] >= .NET 4.5
[ ] before .NET 4.5
[ ] .NET Core 2.0
[ ] .NET Core 2.1
[X] .NET Core 2.2
[ ] .NET Core 3.0

Test Execution Method:

[ ] Visual Studio Test Explorer
[ ] TFS/VSTS/Azure DevOps – Task – PLEASE SPECIFY THE NAME OF THE TASK
[X] Command line – PLEASE SPECIFY THE FULL COMMAND LINE

Command used: dotnet test --filter "(TestCategory=tc:48193)" CroweTaxConnectTests/CroweTaxConnectTests.csproj

<SpecFlow> Section in app.config or content of specflow.json

{
    "bindingCulture":
    {
        "language" :"en-us"
    },
    "language":
    {
        "feature": "en-us"
    },
    "plugins": []
}

Repro Project

Issue Description

Here is the entire output of the test execution:

root@linuxkit-00155dc95309:/DotNetCore# dotnet test --filter "(TestCategory=tc:48193)" CroweTaxConnectTests/CroweTaxConnectTests.csproj
/DotNetCore/CroweTaxConnectApiExtensions/CroweTaxConnectApiExtensions.csproj : warning NU1701: Package 'CroweTaxConnectApiClient 1.0.12' was restored using '.NETFramework,Version=v4.6.1' instead of the project target framework '.NETCoreApp,Version=v2.2'. This package may not be fully compatible with your project. [/DotNetCore/CroweTaxConnectTests/CroweTaxConnectTests.csproj]
/DotNetCore/CroweTaxConnectTests/CroweTaxConnectTests.csproj : warning NU1701: Package 'CroweTaxConnectApiClient 1.0.12' was restored using '.NETFramework,Version=v4.6.1' instead of the project target framework '.NETCoreApp,Version=v2.2'. This package may not be fully compatible with your project.
Build started, please wait...
-> Using specflow.json
/DotNetCore/CroweTaxConnectTests/CroweTaxConnectTests.csproj : warning NU1701: Package 'CroweTaxConnectApiClient 1.0.12' was restored using '.NETFramework,Version=v4.6.1' instead of the project target framework '.NETCoreApp,Version=v2.2'. This package may not be fully compatible with your project.
/DotNetCore/CroweTaxConnectApiExtensions/CroweTaxConnectApiExtensions.csproj : warning NU1701: Package 'CroweTaxConnectApiClient 1.0.12' was restored using '.NETFramework,Version=v4.6.1' instead of the project target framework '.NETCoreApp,Version=v2.2'. This package may not be fully compatible with your project.
Build completed.

Test run for /DotNetCore/CroweTaxConnectTests/bin/Debug/netcoreapp2.2/CroweTaxConnectTests.dll(.NETCoreApp,Version=v2.2)
Microsoft (R) Test Execution Command Line Tool Version 15.9.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
An exception occurred while invoking executor 'executor://specrun/executorV3.0.352': COM interop is not supported on this platform.

Test Run Failed.

Steps to Reproduce

atbe-crowe commented 5 years ago

To add, when I switch my test runner to be MSTest, my tests run successfully. I can't find any documentation on whether specrunner is compatible with linux or not.

SabotageAndi commented 5 years ago

We got the same issue today from another user. I will have a look at it next week. No idea what is going wrong here. We have no COM stuff in SpecFlow+Runner.

atbe-crowe commented 5 years ago

@SabotageAndi yes, it's very odd. I'll try to exclude all of my application code and create a fresh new project that contains the simple out of the box calculator tests and see if I can reproduce the issue.

atbe-crowe commented 5 years ago

@SabotageAndi I have created a reproducible sample project here. You can try to build and run the tests on that project in a linux container and it will fail consistently with the error described at the top of this post.

atbe-crowe commented 5 years ago

@SabotageAndi in addition, I have pushed a Dockerfile that you can build and run to see the issue.

Output:

PS C:\Data\Projects\git\Reproduce-COM-Bug-Specrun> docker run -it repro-specrun-issue
Build started, please wait...
-> Using default config
Build completed.

Test run for /src/CroweTaxConnectTests/bin/Debug/netcoreapp2.2/CroweTaxConnectTests.dll(.NETCoreApp,Version=v2.2)
Microsoft (R) Test Execution Command Line Tool Version 15.9.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
An exception occurred while invoking executor 'executor://specrun/executorV3.0.352': COM interop is not supported on this platform.

Test Run Failed.
Additionally, path to test adapters can be specified using /TestAdapterPath command. Example  /TestAdapterPath:<pathToCustomAdapters>.

SabotageAndi commented 5 years ago

Thanks for it. We are currently looking into it. When we know more, I will inform you.

zeeshankhan0330 commented 5 years ago

@SabotageAndi - Can we get this fix quickly please as this has blocked our CI/CD pipeline.

SabotageAndi commented 5 years ago

@david1995 and I are working on it. Interesting is, that we don't get this COM Error in WSL or in a complete VM. We get it only in the Docker container.

SabotageAndi commented 5 years ago

@atbe-crowe We found a workaround to get the tests executed in Docker container.

We adjusted your srProfile to this:

<?xml version="1.0" encoding="utf-8"?>
<TestProfile xmlns="http://www.specflow.org/schemas/plus/TestProfile/1.5">
  <Settings
      projectName="CroweTaxConnectTests" />
  <Execution
      stopAfterFailures="0"
      testThreadCount="1"
      testSchedulingMode="Random"
      retryFor="Failing"
      retryCount="1" />
  <Environment testThreadIsolation="SharedAppDomain" />
  <Report disable="true" />
  <!-- For collecting by a SpecRun server update and enable the following element. For using the 
      collected statistics, set testSchedulingMode="Adaptive" attribute on the <Execution> element.
    <Server serverUrl="http://specrunserver:6365" publishResults="true" />
  -->
  <TestAssemblyPaths>
    <TestAssemblyPath>CroweTaxConnectTests.dll</TestAssemblyPath>
  </TestAssemblyPaths>
  <DeploymentTransformation>
    <Steps>
      <!-- sample config transform to change the connection string-->
      <!--<ConfigFileTransformation configFile="App.config">
        <Transformation>
          <![CDATA[<?xml version="1.0" encoding="utf-8"?>
                            <configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
                <connectionStrings>
                  <add name="MyDatabase" connectionString="Data Source=.;Initial Catalog=MyDatabaseForTesting;Integrated Security=True" 
                       xdt:Locator="Match(name)" xdt:Transform="SetAttributes(connectionString)" />
                </connectionStrings>
                            </configuration>
                        ]]>
        </Transformation>
      </ConfigFileTransformation>-->
    </Steps>
  </DeploymentTransformation>
</TestProfile>

What this does is:

disables parallel execution
sets test thread isolation mode to SharedAppDomain
disables report generation

The reason for the first and second point is, that we used .NET Standard/Core APIs that are not working on Linux and Mac OS, but in the documentation nothing was mentioning this. :-/ We only found out of this, as we added the API Analyzer. See https://docs.microsoft.com/en-us/dotnet/standard/analyzers/api-analyzer for more information. I can highly recommend to use it.

The report generation needed to be disabled, because this works currently on .NET Framework.

Some of the stuff is quicker to fix, some will take a little time. We will work on this as fast as possible.

zeeshankhan0330 commented 5 years ago

@SabotageAndi - We are using SpecRun for parallel execution and for genrating the report. Can you suggest a Unit test provider which can be used by the time this issue is fixed. Unit test provider which works fine with Specflow 3 on .NetCore + Support parallelization + Generate report + Run inside Docker container. Looking forward to hearing from you.

SabotageAndi commented 5 years ago

We fixed the first issue and with the next version parallel execution in the SharedAppDomain mode should be possible again. Report generation is the next thing we will have a look at.

The new version should be out tomorrow or next Monday.

zeeshankhan0330 commented 5 years ago

@SabotageAndi - Any new build version expected today?

SabotageAndi commented 5 years ago

I hope so. The release pipelines are running.

zeeshankhan0330 commented 5 years ago

@SabotageAndi - with new Version - Test cases are not getting executed. And no error is being displayed either while running it in Docker Linux machine. Here is what I am seeing on the screen:

Command to execute test - dotnet vstest project.dll

The current user does not have a license assigned. This version can be used for evaluation purposes only. For obtaining a registered license please visit http://www.specflow.org/plus. ! Test1 [8ms] ! Test2 [< 1ms] ! Test3 [< 1ms] ! Test4 [< 1ms] Attachments:

SabotageAndi commented 5 years ago

You have so less output, because dotnet test standard loglevel is really low. If you execute it with normal, you get more output. The 3.0.373 version is not yet the version that enables parallel execution in SharedAppDomain again. 3.0.377 will be the version. We found another issue that needed to be fixed for the parallel execution.

zeeshankhan0330 commented 5 years ago

Okay. Lets wait for the next version then. Let me know when can we expect next version

SabotageAndi commented 5 years ago

The release pipeline is still running. So it will take until tomorrow morning until the version is on NuGet.org

SabotageAndi commented 5 years ago

3.0.377 is now uploaded to NuGet.org.

SavitaMaurya commented 5 years ago

@zeeshankhan0330 Did you able to run your tests successfully on Linux ? I am also trying to run the specflow tests using Specflow+Runner 3.0.377.

But first issue is I am not able to register my License on linux! specflow plus license regs error

@SabotageAndi @zeeshankhan0330 Any solution to make it work! I am new to Linux world. Tests are running fine on Windows 10.

SavitaMaurya commented 5 years ago

Hi @SabotageAndi and Team,

May I know when will parallel execution and report generation solution is expected to be out? As we have disabled this to work in Docker

SavitaMaurya commented 5 years ago

Default.txt

Hi @SabotageAndi

My Tests were running fine on docker with the above settings until last week. But today its crashing.

12:54:48 Status: Image is up to date for docker-registry:5000/solar_webshop_automation:staging-latest 12:54:49 Microsoft (R) Test Execution Command Line Tool Version 16.2.0-preview-20190606-02 12:54:49 Copyright (c) Microsoft Corporation. All rights reserved. 12:54:49 12:54:49 Starting test execution, please wait... 12:54:51 ***** 12:54:51 * Evaluation mode * 12:54:51 ***** 12:54:51 12:54:51 The current user does not have a license assigned. 12:54:51 This version can be used for evaluation purposes only. 12:54:51 For obtaining a registered license please visit http://www.specflow.org/plus. 12:55:35 The active test run was aborted. Reason: Test host process crashed : Unhandled Exception: System.Net.Sockets.SocketException: Operation canceled 12:55:35 at System.IO.Pipes.NamedPipeServerStream.gWaitForConnectionAsyncCore|23_0() 12:55:35 at System.Threading.Tasks.TaskToApm.End(IAsyncResult asyncResult) 12:55:35 at TechTalk.SpecFlow.Plus.RPC.Server.<>c__DisplayClass9_0.b0(IAsyncResult ar) 12:55:35 at System.Threading.Tasks.TaskToApm.<>c__DisplayClass3_0.b__0() 12:55:35 at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) 12:55:35 --- End of stack trace from previous location where exception was thrown --- 12:55:35 at System.Threading.Tasks.AwaitTaskContinuation.RunCallback(ContextCallback callback, Object state, Task& currentTask) 12:55:35 --- End of stack trace from previous location where exception was thrown --- 12:55:35 at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) 12:55:35 --- End of stack trace from previous location where exception was thrown --- 12:55:35 at System.Threading.ThreadPoolWorkQueue.Dispatch() 12:55:35 12:55:35 WARNING: Overwriting results file: /data/WebShopTestAutomation/TestResults/./TestResults/report.trx

Has anyone else reported the same error.

Nothing has been changes but keeps on reporting the same error.

Default.srprofile:

<?xml version="1.0" encoding="utf-8"?>
<TestProfile xmlns="http://www.specflow.org/schemas/plus/TestProfile/1.5">
  <Settings projectName="WebShopTestAutomation" />
  <Execution  testThreadCount="1" retryCount="1"/>
  <!-- For collecting by a SpecRun server update and enable the following element. For using the 
      collected statistics, set testSchedulingMode="Adaptive" attribute on the <Execution> element. 
    -->

  <Environment testThreadIsolation="SharedAppDomain" />
  <Report disable="true" />

   <TestAssemblyPaths>
    <TestAssemblyPath>WebShopTestAutomation.dll</TestAssemblyPath>
  </TestAssemblyPaths>
  <Targets>

    <Target name="Headless_Chrome">
      <Filter>Headless_Chrome</Filter>
      <DeploymentTransformationSteps>
        <EnvironmentVariable variable="Test_Browser" value="Headless_Chrome" />
      </DeploymentTransformationSteps>
    </Target>

   </Targets>

</TestProfile>

Can you please help urgently...

SabotageAndi commented 5 years ago

@SavitaMaurya You get the error, because the SpecFlow+Runner can't communicate with it's child processes. If you set the testThreadIsolation to SharedAppDomain the error should go away.

SavitaMaurya commented 5 years ago

@SabotageAndi Thanks for the reply. Its already done and was behaving fine on Linux containter till friday. I have updated the comment by attaching the file.

SabotageAndi commented 5 years ago

Ah, the Markdown was broken. It's fixed now.

Can you check the Default.srProfile file in your bin\debug folder if it has the correct content?

SavitaMaurya commented 5 years ago

Ya.. its same in debug folder..

The Markdown was broken. It's fixed now. >> Can you explain this.. Do I need to take any update?

As I am still facing the same error

SabotageAndi commented 5 years ago

The Markdown was broken. It's fixed now. >> Can you explain this..

If you paste xml directly into a textfield, it will not displayed correctly. You have to surround them with triple ticks ```

@SavitaMaurya did anything else changed in your project? Your config looks good. With the given information, I have currently no idea what could be the cause of the error.

SavitaMaurya commented 5 years ago

@SabotageAndi Ya.. to keep the code repository upto standards I removed the bin, obj,.vs folder and that broked everything. I realized after trying many things the SpecflowPlusRunner folder in bin has to be there to make things work.

So I checked in that again and now everything is working fine.

Thanks for helping.

MatejStraka commented 5 years ago

Hello @SabotageAndi,
I had the same issue with test execution and reports on Mac OS using SpecRunner 3.0.391. I used SharedAppDomain to be able to run tests as suggested in previous comments - this fixed test execution but is there any progress with report generation issue? When I try to generate reports (JSON and HTML) it shows this error: "Could not generate test run result reports. An error occurred in the generator. External program returned exit code '254' Failed generating reports"

SabotageAndi commented 5 years ago

@MatejStraka Sorry, we forgot to update the issue. Reporting in Docker- Containers is working now. But you need to install mono in them. See https://specflow.org/plus/documentation/Reports/ - "Additional Requirements for Non-Windows Users"

zeeshankhan0330 commented 5 years ago

Hi Andy,

Is the report available for community version as well?

On Mon, Oct 21, 2019, 8:00 PM Andreas Willich notifications@github.com wrote:

@MatejStraka https://github.com/MatejStraka Sorry, we forgot to update the issue. Reporting in Docker- Containers is working now. But you need to install mono in them. See https://specflow.org/plus/documentation/Reports/ - "Additional Requirements for Non-Windows Users"

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/techtalk/SpecFlow/issues/1662?email_source=notifications&email_token=AEOYUKI5OR7TQEDPYQXDBUTQPW4H7A5CNFSM4ICLOK3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEB2QWKY#issuecomment-544541483, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEOYUKIT2YEVBNYSPJTMAV3QPW4H7ANCNFSM4ICLOK3A .

SabotageAndi commented 5 years ago

@zeeshankhan0330 There are no reports in the open source project currently. See details: https://github.com/techtalk/SpecFlow/issues/1036

SabotageAndi commented 5 years ago

So, it took longer than we wanted, but the SpecFlow+Runner is now working in Docker with Reports and Parallel Execution in different processes. You have to update to 3.1.21-beta.

You can find an example for it here: https://github.com/techtalk/SpecFlow.Plus.Examples/tree/master/DockerExample Be aware that you need mono installed in your Docker image that you can generate reports.

The documentation is now yet fully there.

Please open new issues if you find issues with the Docker support.

lock[bot] commented 4 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

SpecFlowOSS / SpecFlow