search

reportportal / agent-net-vstest

Logger extension for VsTest
Apache License 2.0
8 stars 10 forks source link

How to integrate ReportPortal with .net core with Visual Studio Test Explorer #27

Open ghost opened 5 years ago

ghost commented 5 years ago

Hi, I want to migrate my tests from .net framework to .net core and I do not understand how to integrate the report portal with my tests. I read readme file and there it is somehow incomprehensibly described flow for .net core. Could you explain me it possible, and if yes, then how we can do it. Thanks!

nvborisenko commented 5 years ago

Actually .netcore is possible only with vstest runner and NUnit3TestAdapter nuget package. Reporting results to RP will be possible when new version of NUnit3TestAdapter package will be released (v3.14.0). It is not officially released yet, we are in awaiting of this version.

You can ask here what is planned date of release. Last time when I asked was end of April, might be they forgot to release it, not sure.

ghost commented 5 years ago

@nvborisenko, Thanks for your answer, but I'm trying to run my .net core tests via vstest runner and get next error: [7/5/2019 11:49:01 AM Error] Microsoft.VisualStudio.TestPlatform.Common.Exceptions.InvalidLoggerException: Could not find a test logger with AssemblyQualifiedName, URI or FriendlyName 'ReportPortal'. at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.TestLoggerManager.Initialize(String runSettings) at Microsoft.VisualStudio.TestPlatform.Client.TestPlatform.CreateTestRunRequest(IRequestData requestData, TestRunCriteria testRunCriteria, TestPlatformOptions options) at Microsoft.VisualStudio.TestPlatform.CommandLine.TestPlatformHelpers.TestRequestManager.RunTests(IRequestData requestData, TestRunCriteria testRunCriteria, ITestRunEventsRegistrar testRunEventsRegistrar, TestPlatformOptions options) at Microsoft.VisualStudio.TestPlatform.CommandLine.TestPlatformHelpers.TestRequestManager.RunTests(TestRunRequestPayload testRunRequestPayload, ITestHostLauncher testHostLauncher, ITestRunEventsRegistrar testRunEventsRegistrar, ProtocolConfig protocolConfig) at Microsoft.VisualStudio.TestPlatform.Client.DesignMode.DesignModeClient.<>c__DisplayClass18_0.<StartTestRun>b__0() but if I'm using dotnet test, than all works fine.

P.S my *.runsettings file: image

it looks like <TestAdaptersPaths>.</ TestAdaptersPaths> doesn't connect needed adapter for ReportPortal. I assume that the studio does not run tests from the working directory of the project. maybe you met this?

nvborisenko commented 5 years ago

To do it, what you wanted, you need use https://github.com/reportportal/agent-net-vstest agent.

If you already use it please share the list of installed nuget packages and the version of Visual Studio. It works for me with v16.1.1. Not sure, this magic is supported beginning v15.9.

ghost commented 5 years ago

@nvborisenko image image

nvborisenko commented 5 years ago

Please share your config file. I reproduced your issue if url parameter is incorrect.

ghost commented 5 years ago

@nvborisenko image

nvborisenko commented 5 years ago

Does dotnet test -l:ReportPortal work properly?

ghost commented 5 years ago

@nvborisenko Yes, it works with dotnet test -l:ReportPortal, but I'm trying to run tests from visual studio with Report portal integration.

nvborisenko commented 5 years ago

@OlegYanushkevich please review my Visual Studio settings: image

If you have the same settings - I have no idea what is wrong.

ghost commented 5 years ago

@nvborisenko same settings for my Visual Studio:) I do not know what could be the reason. Thanks for your help.

nvborisenko commented 5 years ago

Try to change logging level to diagnostical on the screenshot above. Might we see better error message.

nvborisenko commented 5 years ago

@OlegYanushkevich one more idea: verify that you don't have specified environment variables starting with REPORTPORTAL_.

suszi88 commented 5 years ago

I think I had to install ReportPortal.Shared as I was getting same error, although I was not using nUnit

antonmalch commented 4 years ago

@suszi88 , @OlegYanushkevich, @nvborisenko Hi Guys. Wanted to ask ) In the end you solved the problem? Because Looks like i have the same issue now.
pic1 pic2

[03/03/2020 14:13:02 Diagnostic] RunSettings Content:

. C:\Users\antonm\Source\Repos\UnitTestProject1\TestResults C:\Users\antonm\Source\Repos\UnitTestProject1\ X86 Framework45 > Ran from Visual Studio Test ExplorerC:\Users\antonm\Source\Repos\UnitTestProject1\UnitTestProject1

[03/03/2020 14:13:02 Diagnostic] Value of UseSpecifiedAdapterLocations as specified in Tools -> Options: True [03/03/2020 14:13:02 Diagnostic] Test framework could not be determined for project: C:\Users\antonm\Source\Repos\UnitTestProject1\UnitTestProject1\bin\Debug\netcoreapp2.1\UnitTestProject1.dll because reference path is null for at least one reference: Microsoft.NETCore.App [03/03/2020 14:13:02 Diagnostic] Value of FallbackToGlobalVsix as specified in Tools -> Options: True [03/03/2020 14:13:02 Diagnostic] Adapter location settings being used after evaluation: UseSpecifiedAdapterLocations = False, SkipDefaultAdapters = False, AreAllTestContainersCSharpOrVBProjectsOnly = True [03/03/2020 14:13:04 Diagnostic] Discovering tests in C:\Users\antonm\Source\Repos\UnitTestProject1\UnitTestProject1\ReportPortal.cs [03/03/2020 14:13:04 Diagnostic] RockSteady initialized with the following extensions: C:\Users\antonm.nuget\packages\mstest.testadapter\1.3.2\build_common\Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll

[03/03/2020 14:13:05 Diagnostic] Final RunSettings for the current TestRunCriteria:

. C:\Users\antonm\Source\Repos\UnitTestProject1\TestResults C:\Users\antonm\Source\Repos\UnitTestProject1\ X86 FrameworkCore10 False > Ran from Visual Studio Test ExplorerC:\Users\antonm\Source\Repos\UnitTestProject1\UnitTestProject1

[03/03/2020 14:13:08 Error] Microsoft.VisualStudio.TestPlatform.Common.Exceptions.InvalidLoggerException: Could not find a test logger with AssemblyQualifiedName, URI or FriendlyName 'ReportPortal'. at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.TestLoggerManager.Initialize(String runSettings) at Microsoft.VisualStudio.TestPlatform.Client.TestPlatform.CreateTestRunRequest(IRequestData requestData, TestRunCriteria testRunCriteria, TestPlatformOptions options) at Microsoft.VisualStudio.TestPlatform.CommandLine.TestPlatformHelpers.TestRequestManager.RunTests(IRequestData requestData, TestRunCriteria testRunCriteria, ITestRunEventsRegistrar testRunEventsRegistrar, TestPlatformOptions options) at Microsoft.VisualStudio.TestPlatform.CommandLine.TestPlatformHelpers.TestRequestManager.RunTests(TestRunRequestPayload testRunRequestPayload, ITestHostLauncher testHostLauncher, ITestRunEventsRegistrar testRunEventsRegistrar, ProtocolConfig protocolConfig) at Microsoft.VisualStudio.TestPlatform.Client.DesignMode.DesignModeClient.<>c__DisplayClass19_0.b__0() [03/03/2020 14:13:08 Diagnostic] Cancel requested, actions left in queue 2, exiting... [03/03/2020 14:13:08 Diagnostic] VirtualReadOnlyTestDataStore.OperationStateChanged State=TestExecutionCanceling, operationInProgress=False [03/03/2020 14:13:08 Diagnostic] TestDiscoveryStats.OperationStateChanged State=TestExecutionCanceling, InProgress=False [03/03/2020 14:13:08 Diagnostic] VirtualReadOnlyTestDataStore.OperationStateChanged State=OperationSetFinished, operationInProgress=False [03/03/2020 14:13:08 Diagnostic] TestDiscoveryStats.OperationStateChanged State=OperationSetFinished, InProgress=False [03/03/2020 14:13:08 Diagnostic] Run finished using 'InMemoryUnitTestWriter' [03/03/2020 14:13:08 Informational] ========== Run test finished: 0 run (0:00:05.3267755) ========== [03/03/2020 14:13:08 Diagnostic] VirtualReadOnlyTestDataStore.OperationStateChanged State=TestExecutionCancelAndFinished, operationInProgress=False [03/03/2020 14:13:08 Diagnostic] TestDiscoveryStats.OperationStateChanged State=TestExecutionCancelAndFinished, InProgress=False

suszi88 commented 4 years ago

@antonmalch I installed the ReportPortal.Shared nuget and it solved the issue. I see that you do not have it

nvborisenko commented 4 years ago

ReportPortal.Shared package is installed automatically, it's ok that you don't see it in the list of installed packages.

@antonmalch could you please share your test solution and how you execute tests?

antonmalch commented 4 years ago

@nvborisenko It's new Project (for testing Report Portal) Pic3

suszi88 commented 4 years ago

@antonmalch just for the record, for me running from vs is not working, it will only integrate with report portal while running from cmd, but it's my case, hope you will solve your issue

antonmalch commented 4 years ago

Hi, @suszi88 , ok, understood, thank you

nvborisenko commented 4 years ago

Visual Studio 2017?

antonmalch commented 4 years ago

Hi, yes pic4

nvborisenko commented 4 years ago

@antonmalch do you have any ReportPortal.*.log files in the directory where sln or csproj file is located, or near with tests assembly?

antonmalch commented 4 years ago

@nvborisenko Good morning. Looks like no. I have not ReportPortal.*.log

image image

nvborisenko commented 4 years ago

Hey @antonmalch @suszi88 @OlegYanushkevich I was able to reproduce this issue and found out what was a problem.

Please try to use v3.0.0-beta.2 version of the package. I suggest to restart VS after upgrading to let VS reload Logger.

Note: this package is only for the latest ReportPortalv5.

image

@antonmalch you will get errors, but it will be separate issue (incorrect url parameter in config). Please share your ReportPortal.*.log files near with sln.

Guys, please report back whether it fixes the issue.

antonmalch commented 4 years ago

Hi,

v3.0.0-beta.2 version of the package - done

But I can't find ReportPortal.*.log files. This file(s) should installed automatically?

pic5

Pic6 pic7

nvborisenko commented 4 years ago

Will be appeared after tests execution.

antonmalch commented 4 years ago

@nvborisenko , I have execute test (from VS) few times and nothing. Still can't find ReportPortal.*.log files.

pic11

pic12

pic13

pic14

[04/03/2020 15:21:14 Diagnostic] About to Enqueue operation 'RunFromContextOperation', hashcode:51164618 [04/03/2020 15:21:14 Diagnostic] Enqueue operation 'RunFromContextOperation', hashcode:51164618 [04/03/2020 15:21:14 Diagnostic] Operation left in the the queue: 1 [04/03/2020 15:21:14 Diagnostic] 'RunFromContextOperation', hashcode:51164618 [04/03/2020 15:21:14 Diagnostic]

[04/03/2020 15:21:14 Diagnostic] Processing Queue ..... [04/03/2020 15:21:14 Diagnostic] Operation Dequeue : 'RunFromContextOperation' [04/03/2020 15:21:14 Diagnostic] VirtualReadOnlyTestDataStore.OperationStateChanged State=OperationSetStarted, operationInProgress=False [04/03/2020 15:21:14 Diagnostic] TestDiscoveryStats.OperationStateChanged State=OperationSetStarted, InProgress=False [04/03/2020 15:21:14 Diagnostic] VirtualReadOnlyTestDataStore.OperationStateChanged State=ChangeDetectionStarting, operationInProgress=True [04/03/2020 15:21:14 Diagnostic] TestDiscoveryStats.OperationStateChanged State=ChangeDetectionStarting, InProgress=False [04/03/2020 15:21:14 Diagnostic] We programmatically start the build in TestWindowHost.UpdateContainer... [04/03/2020 15:21:14 Diagnostic] Adding the Project in Project Map for Guid 0d4c4dfd-517d-47b6-953e-15a0869c4686 [04/03/2020 15:21:14 Diagnostic] Time taken for fetching Roslyn supported projects: 0 milliseconds. [04/03/2020 15:21:14 Diagnostic] After we await the build that programmatically started in TestWindowHost.UpdateContainer... [04/03/2020 15:21:14 Diagnostic] Adding the Project in Project Map for Guid 0d4c4dfd-517d-47b6-953e-15a0869c4686 [04/03/2020 15:21:14 Diagnostic] TestContainer update (build) complete : 387 ms [04/03/2020 15:21:14 Diagnostic] Adding the Project in Project Map for Guid 0d4c4dfd-517d-47b6-953e-15a0869c4686 [04/03/2020 15:21:14 Diagnostic] Adding the Project in Project Map for Guid 0d4c4dfd-517d-47b6-953e-15a0869c4686 [04/03/2020 15:21:14 Diagnostic] Test framework could not be determined for project: C:\Users\antonm\Source\Repos\UnitTestProject1\UnitTestProject1\bin\Debug\netcoreapp2.1\UnitTestProject1.dll because reference path is null for at least one reference: Microsoft.NETCore.App [04/03/2020 15:21:14 Diagnostic] DiscoveryOperation FinishedChangedCotainers, changed container count is 0 [04/03/2020 15:21:14 Diagnostic] VirtualReadOnlyTestDataStore.OperationStateChanged State=ChangeDetectionFinished, operationInProgress=False [04/03/2020 15:21:14 Diagnostic] TestDiscoveryStats.OperationStateChanged State=ChangeDetectionFinished, InProgress=False [04/03/2020 15:21:14 Informational] Executing test method 'UnitTestProject1.ReportPortal.Test_1111_TestReportPortal' [04/03/2020 15:21:14 Diagnostic] VirtualReadOnlyTestDataStore.OperationStateChanged State=TestExecutionStarting, operationInProgress=False [04/03/2020 15:21:14 Diagnostic] TestDiscoveryStats.OperationStateChanged State=TestExecutionStarting, InProgress=False [04/03/2020 15:21:14 Informational] ------ Run test started ------ [04/03/2020 15:21:14 Diagnostic] VirtualReadOnlyTestDataStore.OperationStateChanged State=TestExecutionStarted, operationInProgress=False [04/03/2020 15:21:14 Diagnostic] TestDiscoveryStats.OperationStateChanged State=TestExecutionStarted, InProgress=False [04/03/2020 15:21:14 Diagnostic] RunSettings Content:

. C:\Users\antonm\Source\Repos\UnitTestProject1\TestResults C:\Users\antonm\Source\Repos\UnitTestProject1\ X86 Framework45 > Ran from Visual Studio Test ExplorerC:\Users\antonm\Source\Repos\UnitTestProject1\UnitTestProject1

[04/03/2020 15:21:14 Diagnostic] Value of UseSpecifiedAdapterLocations as specified in Tools -> Options: True [04/03/2020 15:21:14 Diagnostic] Value of FallbackToGlobalVsix as specified in Tools -> Options: True [04/03/2020 15:21:14 Diagnostic] Adapter location settings being used after evaluation: UseSpecifiedAdapterLocations = False, SkipDefaultAdapters = False, AreAllTestContainersCSharpOrVBProjectsOnly = True [04/03/2020 15:21:15 Diagnostic] Final RunSettings for the current TestRunCriteria:

. C:\Users\antonm\Source\Repos\UnitTestProject1\TestResults C:\Users\antonm\Source\Repos\UnitTestProject1\ X86 FrameworkCore10 False > Ran from Visual Studio Test ExplorerC:\Users\antonm\Source\Repos\UnitTestProject1\UnitTestProject1

[04/03/2020 15:21:16 Diagnostic] Discovering tests in C:\Users\antonm\Source\Repos\UnitTestProject1\UnitTestProject1\ReportPortal.cs [04/03/2020 15:21:16 Diagnostic] VirtualReadOnlyTestDataStore.OperationStateChanged State=OperationSetFinished, operationInProgress=False [04/03/2020 15:21:16 Diagnostic] TestDiscoveryStats.OperationStateChanged State=OperationSetFinished, InProgress=False [04/03/2020 15:21:16 Error] Microsoft.VisualStudio.TestPlatform.Common.Exceptions.InvalidLoggerException: Could not find a test logger with AssemblyQualifiedName, URI or FriendlyName 'ReportPortal'. at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.TestLoggerManager.Initialize(String runSettings) at Microsoft.VisualStudio.TestPlatform.Client.TestPlatform.CreateTestRunRequest(IRequestData requestData, TestRunCriteria testRunCriteria, TestPlatformOptions options) at Microsoft.VisualStudio.TestPlatform.CommandLine.TestPlatformHelpers.TestRequestManager.RunTests(IRequestData requestData, TestRunCriteria testRunCriteria, ITestRunEventsRegistrar testRunEventsRegistrar, TestPlatformOptions options) at Microsoft.VisualStudio.TestPlatform.CommandLine.TestPlatformHelpers.TestRequestManager.RunTests(TestRunRequestPayload testRunRequestPayload, ITestHostLauncher testHostLauncher, ITestRunEventsRegistrar testRunEventsRegistrar, ProtocolConfig protocolConfig) at Microsoft.VisualStudio.TestPlatform.Client.DesignMode.DesignModeClient.<>c__DisplayClass19_0.b__0() [04/03/2020 15:21:16 Diagnostic] Cancel requested, actions left in queue 2, exiting... [04/03/2020 15:21:16 Diagnostic] VirtualReadOnlyTestDataStore.OperationStateChanged State=TestExecutionCanceling, operationInProgress=False [04/03/2020 15:21:16 Diagnostic] TestDiscoveryStats.OperationStateChanged State=TestExecutionCanceling, InProgress=False [04/03/2020 15:21:16 Diagnostic] Run finished using 'InMemoryUnitTestWriter' [04/03/2020 15:21:16 Informational] ========== Run test finished: 0 run (0:00:01.7459409) ========== [04/03/2020 15:21:16 Diagnostic] VirtualReadOnlyTestDataStore.OperationStateChanged State=TestExecutionCancelAndFinished, operationInProgress=False [04/03/2020 15:21:16 Diagnostic] TestDiscoveryStats.OperationStateChanged State=TestExecutionCancelAndFinished, InProgress=False

nvborisenko commented 4 years ago

@antonmalch And have you restarted Visual Studio?

antonmalch commented 4 years ago

@nvborisenko Yes

nvborisenko commented 4 years ago

I also see issues on clean environment with VS2017 and .netcore2.1, sometimes I saw different issues, It depends on @DzmitryHumianiuk how much we want to invest to resolve the issue.

DzmitryHumianiuk commented 4 years ago

@nvborisenko it's hard for to evaluate something in .net world. So i should rely on you here.

jsingh-qualitrol commented 4 years ago

@nvborisenko @antonmalch have you got solution to the problem? I am also getting the same error with latest v3.0.0-beta.3.Please help

nvborisenko commented 4 years ago

@jsingh-qualitrol VS 2017?

jsingh-qualitrol commented 4 years ago

No i am using VS 2019

nvborisenko commented 4 years ago

https://github.com/reportportal/agent-net-vstest#visual-studio-test-explorer

did you try it?

jsingh-qualitrol commented 4 years ago

yes i followed the same steps mentioned in the link. @nvborisenko

nvborisenko commented 4 years ago

https://github.com/reportportal/example-net-vstest please play with this pre-configured project.

nvborisenko commented 4 years ago

@jsingh-qualitrol please share a project you have played with.

nvborisenko commented 4 years ago

Reproduced it again on my side. Working solution is specifying full path to output folder (bin\Debug) in runsettings file, TestAdaptersPaths property.

BilalKhan199 commented 4 years ago

Update: first time you have to provide full path, then it works without providing full path, I'm using VS2019

nvborisenko commented 3 years ago

And finally the root cause of the issue is relative path specified in runsettings file. Official way is to use . (current directory) as a location where VS (or any other test runner) should search for test adapter (test logger). Since this is relative path, the question is relative to what.

When we execute tests in Visual Studio, then my finding is relative path to Visual Studio directory where VS was run from.

If I start VS from Windows task bar, the current directory is C:\\Program Files\\.... And, VS runner cannot find test logger. But if I open solution file in File Explorer (double click on *.sln) then current directory for VS is the directory where *.sln file is located. In this case VS is able to find test logger in project's bin/Debug directory.

Thus, for all who is experiencing this issue, please open your solution file using double click in File Explorer.

This issue cannot be considered as resolved, let's see what MS swill say.