Open frederik-h opened 5 years ago
full .net vstest and .net core vstest have a difference how these tools find adapters in different versions of .net SDK.
But your project file contains package reference to <PackageReference Include="TeamCity.VSTest.TestAdapter" Version="1.0.10" />
. This package configures the test adapters path and the logger automatically. So try avoiding these arguments --test-adapter-path
and --logger
in your command line.
For the case under TeamCity the command line could look like dotnet test /p:TargetFramework=net471
or just dotnet test
to run all tests.
For the case not under TeamCity you cold add an additional argument like/p:TEAMCITY_VERSION=2018
to notify TeamCity logger to be active this time:
dotnet test /p:TargetFramework=net471 /p:TEAMCITY_VERSION=2018
Thanks for your quick reply. The perhaps most important thing first:
Invoking the tests with /p:TEAMCITY_VERSION=2018
yields the desired result for both SDK versions. Thanks!
Some things that don't work:
1) Running dotnet test
with the "command line" runner (with the TC adapter as a package reference in the project file) within TeamCity executes the tests but does not produce the output format expected by TC. Same result on my local machine.
2) Running dotnet test
using the "command line" runner within TeamCity with the explicit --test-adapter-path
(pointing to a path that contains the adapter DLL) and --logger
arguments and without the Package Reference does not find the tests. (We have SDK 2.1.301 on our build agents.)
3) Running the tests with the ".NET CLI (dotnet)" runner and the "Command" "test" does not execute the tests (same behavior for "vstest"). But, if you look at the command line in the build logs, this does the same as in the previous step.
Perhaps the TEAMCITY_VERSION
parameter could be documented more prominently? Of course, it would be even better, to find out why it is necessary and make the adapter behave as expected without it.
I would also expect the ".NET CLI (dotnet)" runner to fail if no tests were discovered (you don't add a test build step if you do not have any tests to run), but that should probably be discussed elsewhere.
@frederik-h I would like to mention that the additional argument /p:TEAMCITY_VERSION=2018
is not required under TeamCity, TeamCity makes it automatically. For local test runs on your computer TeamCity output is hidden by design because of it does not suppose TeamCity integration. For the docker scenario there is a small trick.
@frederik-h
Running dotnet test with the "command line" runner (with the TC adapter as a package reference in the project file) within TeamCity executes the tests but does not produce the output format expected by TC. Same result on my local machine.
Is it possible to share some screenshots and build log files?
Running dotnet test using the "command line" runner within TeamCity with the explicit --test-adapter-path (pointing to a path that contains the adapter DLL) and --logger arguments and without the Package Reference does not find the tests. (We have SDK 2.1.301 on our build agents.)
Unfortunately --test-adapter-path
does not allow to use several paths, but it is required because vstest should find the specific test adapter and TeamCity adapter. But there is an environment variable VSTestTestAdapterPath
. It allows to combine several paths using ;
(begining from the some version of .net SDK).
Running the tests with the ".NET CLI (dotnet)" runner and the "Command" "test" does not execute the tests (same behavior for "vstest"). But, if you look at the command line in the build logs, this does the same as in the previous step.
Could you add details?
Could you describe your scenario in details? I mean:
dotnet --info
):warning: If there is some confidential information just create an issue here and add comments/artifacts for JetBrains team only.
@NikolayPianikov It is not clear to me for which runner this should work? If I use the ".NET CLI (dotnet)" runner, it does not work for me. It works if I replace this build step by a command line build step which invokes dotnet test manually with the version parameter. By the way, the version line of the UI says that we are running "TeamCity Enterprise 2018.1.1 (build 58406)".
What you said regarding local test runs is also not consistent with the animated gif at the project's github page which shows dotnet test being invoked without the version parameter producing the service messages.
@frederik-h Unfortunately vstest has this issue, but the vstest team does not want to fix it. May be it is this issue. Thus you could update your TeamCity to the latest version because we have tried to apply some workarounds here. Or just manually set the verbosity level to normal
for test
step in the dotnet CLI plugin or in the command line runner.
Also we have the related issue here
@NikolayPianikov It is probably not related to those issues since the output of the dotnet test command in the build log of the dotnet cli build step explicitly says that "No test is available in ..." (for target framework net462, with SDK 2.1.301). I cannot update anything on the build agents, but I have several workarounds that I can use until we get updated TC and dotnet SDK versions. And the same behavior with my minimal test project that I described above goes aways with SDK 2.1.401. Hence, I can use the workarounds for now and wait for the updates to be installed on our agents.
To me, issues #10 and #11 seem to be related to this issue.
In scenarios where you have mixed test projects, eg XUnit and Nunit, we had added the Logger console and verbosity commands to make the Xunit output visible and were confused why the NUnit output would not appear when we added the Adapter nuget package.... The gold part from the above threads is you can't just add it to one project if you are providing the logger command... The correct way is to stop overriding the logger for dotnet test and just add the VSTest.TestAdapter to all projects .. works like a charm
I have some trouble with test projects on TeamCity for which the tests stopped executing at some point (TC update, dotnet SDK update?!). Unfortunately, this did not cause the dotnet build step to fail and hence it went unnoticed for quite some time. I have traced back the problems to the TeamCity.VSTest.TestAdapter. As a minimal example with similar behavior, consider a project with the following .csproject file
which contains a single source file (whose content is not really important, but included for completeness):
If I run the tests with my local dotnet core SDK version 2.1.401, I get the expected results with both the netcoreapp2.0 and the net471 tests.
I get the same result if I run with
/p:TargetFramework=net471
.Now, if I set the SDK version to 2.1.301 (the version on our build agents) with
dotnet new globaljson --sdk-version 2.1.301
, the netcoreapp2.0 tests still run with the same result. But for net471 , I get:If I set the test adapter path by
--test-adapter-path:.
, the net471 build works as well. But for my actual project, using this variant does not work at all, whereas the variant with the explicit paths shows the same behavior as described here.