Support for .Net Core projects

[CharliePoole]

@galvesribeiro commented on Sat Jan 14 2017

Visual Studio is throwing this:

------ Discover test started ------
NUnit Adapter 3.6.0.0: Test discovery starting
Error: Unable to get runner for this assembly. Check installation, including any extensions.
FileNotFoundException: Could not load file or assembly 'nunit.framework' or one of its dependencies. The system cannot find the file specified.
Dependent Assembly nunit.framework of C:\Users\gutem\documents\visual studio 2017\Projects\ClassLibrary1\ClassLibrary1\bin\Debug\netstandard1.6\ClassLibrary1.dll not found. Can be ignored if not a NUnit project.
NUnit Adapter 3.6.0.0: Test discovery complete
========== Discover test finished: 0 found (0:00:00.655) ==========

This is the sample .csproj:

<Project Sdk="Microsoft.NET.Sdk" ToolsVersion="15.0">
  <PropertyGroup>
    <TargetFramework>netstandard1.6</TargetFramework>
  </PropertyGroup>
  <ItemGroup>
    <Compile Include="**\*.cs" />
    <EmbeddedResource Include="**\*.resx" />
  </ItemGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.0.0-preview-20170113-02" />
    <PackageReference Include="NETStandard.Library" Version="1.6.1" />
    <PackageReference Include="NUnit" Version="3.6.0" />
    <PackageReference Include="NUnit3TestAdapter" Version="3.6.0" />
  </ItemGroup>
</Project>

With this sample test:

[TestFixture]
    public class Class1
    {
        string teststring = "";

        [SetUp]
        public void SetUp()
        {
            teststring = "AAA";
        }

        [Test]
        public void TestDemo()
        {
            Assert.True(teststring.Equals("AAA"));
        }
    }

Do we need to do anything else?

Thank you.

---

@jnm2 commented on Sat Jan 14 2017

@CharliePoole I wonder if this is https://github.com/nunit/nunit3-vs-adapter/issues/296?

@galvesribeiro Can you see if the NUnit console runner runs the tests properly?

---

@galvesribeiro commented on Sat Jan 14 2017

@jnm2 what should I do to use the console runner on .net standard projects?

---

@JustinRChou commented on Sat Jan 14 2017

Seems like it. NUnit Adapter 3.6.1.0 has the same issue.

---

@jnm2 commented on Sat Jan 14 2017

@galvesribeiro Download the zip from https://github.com/nunit/nunit-console/releases and run nunit3-console "path\to\test\assembly.dll" at the command line.

---

@jnm2 commented on Sat Jan 14 2017

@JustinRChou can we make these new-style .csproj's part of the VS adapter tests, both .NET Standard and .NET Framework? A single project can multitarget and produce both binaries.

[rprouse]

@galvesribeiro this is a known issue. The NUnit Engine does not currently support .NET Core projects and the dotnet-test-nunit adapter doesn't work with the new .NET Core CSPROJ format. Microsoft decided to drop support for the various dotnet-test-* style adapters in favor of the old style visual studio adapters and we are still playing catchup.

[galvesribeiro]

@rprouse that is strange... xUnit still working just fine since the beginning of .Net Core with DNX. I though when you released 3.6 which is supposed to support .Net Standard 1.6 projects it should work. My mistake...

[rprouse]

@galvesribeiro xUnit has the advantage that the .NET core team is using xUnit, so they were updated first 😄

We released the .NET Standard version of the framework as a first step towards runner support. As a holdover until we are ready, you can make your tests self-executable using NUnitLite.

[galvesribeiro]

@rprouse right. The only problem with that is that I can't use VS to run/debug the tests... Ok, I understood...

Thanks, please close the issue.

[jnm2]

@rprouse Are you planning to support .NET Standard 1.4 or 2.0? The reason I ask is that I have .NET Standard DLLs that I want to test on the .NET Framework, not .NET Core, runtime.

[rprouse]

@jnm2 we will probably replace the PCL project with .NET Standard 1.0 or 1.3 and we may add 2.0 support if needed. Remember, you can always use a lower version of .NET Standard NUnit framework in a higher level project.

[CharliePoole]

Lost the internet for a while or I would have commented sooner...

The 3.6 release of the adapter is not the new one we are working on. It came out a few months ago, before we had any support for .NET Standard. I'm working on finishing the 3.7 release, but that isn't going to solve the problem either.

Runners (like the console, gui or VS adapter) depend on the nunit engine to load and run tests. As soon as we have an engine that supports .NET standard, we'll be able to make the runners support it. Since we release the console together with the engine, that support will be immediate. For the adapter, it won't happen until (a) there is an engine with support and (b) the adapter is updated to use that engine.

We won't close this because this is support we want to provide. But I'm marking it as blocked until there is engine support for it.

[jnm2]

@CharliePoole when you have a project that targets netcoreapp10;net45;net46 it's obvious what to do- run each of the three output binaries against its target TFM. (The reason you might want to test against both 4.5 and 4.6 is quirking.)

However, if one of a project's targets is netstandard14, what runtime is that output binary tested against? .NET Core and .NET Framework are both common candidates and there are other runtimes besides. It would be awesome if I could compile to a single .NET Standard binary and do all my testing off that same binary for every platform (.NET Core, .NET Framework, Xamarin, UWP, whatever).

Will there be a way to specify to NUnit which runtimes to test against, or will it test against all of them that it can?

[CharliePoole]

@jnm2 It's less obvious how to do it. 😄

We are a project that needs to test against multiple runtimes and we do it by replicating test projects. It would be cool to be able to do it in an easier way, but first we have to be able to run the net standard tests at all. IOW, I'm trying to figure out how to get into orbit and you're ready to land on Mars. 😄

NUnit - up to now - makes no decisions on this. It just runs against what you tell it to run against. I don't think that automatically running against multiple runtimes can be an engine feature. It has to be in the runner. The reason I say that is that runners are likely to get pretty confused if they tell the engine to run an assembly and it runs it three times. The runner has to be ready to receive what the engine produces. OTOH, the engine could introduce a setting that allows passing multiple platforms as we now pass in a single platform.

Currently, if you have a .NET 2.0 assembly (to use old technology as an example) you can tell the engine to run it under .NET 4.0. If you don't tell it, it will use 2.0 if available, otherwise the lowest compatible runtime available. Hypothetically, we could pass in an argument something like "net-2.0+net-4.0+net-4.5" and expect it to be run three times. Something similar could be done for portable or net standard tests - it's just that nobody ever thought of it before. Feel free to introduce it as an issue so we don't forget it and also to keep this issue clean.

[galvesribeiro]

I'm not sure if run multiple times against multiple frameworks is something that worth... The principle of netstandard is to support the same API surface between multiple TFMs so, I believe if you target your library to netstandard even if your main facus are net45 framework would be the best solution. Multitargeting on netstandard tooling is supported but is not encouraged since it goes against its purpose.

[galvesribeiro]

But like @CharliePoole said, if you REALLY want to test it that way, make your library to multitarget and create multiple test projects and if they are all have the same set of test code, put it in a shared test project which is referenced on each test project for that specific platform.

[jnm2]

@galvesribeiro The way I really want to test is to produce a single netstandard output and have it run against various versions of .NET Core and .NET Framework. I don't want to multitarget if I don't have to. Then again, there will be scenarios where I need to target netstandard1.4;net40 and I won't have an option.

[jnm2]

@CharliePoole

Feel free to introduce it as an issue so we don't forget it and also to keep this issue clean.

I'm happy to. Not sure where to put the issue since it affects more than one runner.

Let's just take a common case, where I'm forced to target netstandard14;net40. Which runtimes will be used to test the two binaries? What would be ideal for me is for the net40 output binary to be tested against net40 and for the netstandard14 output binary to be tested against the earliest version of .NET Core (1.0) and the earliest version of .NET Framework (4.6.1) that support it. However, someone else may want to also test the netstandard binary against UWP 10 or Xamarin vNext or the latest .NET Core.

[CharliePoole]

@jnm2 I think it has to be done in the engine and console first. It can be added elsewhere after that's done. Each runner would get a separate issue because we are trying to treat each runner as a separate project but there's no point in creating (for example) an adapter issue until there's a feature in the engine that allows the adapter to do this.

Currently the engine runs tests once, under the specified framework or an automatically selected framework. Since the engine doesn't work with multi-targeted assemblies, it can always automatically select a single framework. I think the initial option needed to power all of this is the ability to tell the engine to run tests under multiple frameworks. Everything comes from that.

[jnm2]

I understood you to be saying it would not be done in the engine:

I don't think that automatically running against multiple runtimes can be an engine feature. It has to be in the runner. The reason I say that is that runners are likely to get pretty confused if they tell the engine to run an assembly and it runs it three times.

But since I quietly disagreed with that earlier, I'll just stay out of the way now 😆

[CharliePoole]

If we want the tests to run three times in one Run call, then the engine has to do that. What I wrote earlier was that the engine can't do that automatically because it doesn't know whether the runner calling it can handle the triple output. The engine isn't intended to put demands on runners, but to accept commands from them.

OTOH, in order for a runner to command the engine to do it, we have to implement the capability in the engine. Does that make more sense?

[CharliePoole]

BTW,,, I missed your earlier disagreement... still can't find it.

[jnm2]

That makes perfect sense.

BTW,,, I missed your earlier disagreement... still can't find it.

quietly internally. Everything looks good. I'll try not to confuse anything from this point forward. 🙄

[rprouse]

Now that VSTest has been open sourced, I have been reading through the documentation and wanted to add some notes to this issue in preparation for working on it. This issue is blocked until we have .NET Core support in the engine, but how we implement that support in the engine may be driven by how we will consume it in this adapter.

For example, AFAIK, this adapter runs tests in-process and does not use the agents. The new dotnet test command line will also use this adapter to run tests at the command line. That will be the documented way to run tests from the command line, so do we need to also add support to the console runner?

I've been looking at how MSTest and xUnit package their adapters now to support .NET Core. It is very similar to other NuGet packages with platform specific versions of the *.TestAdapter.dll in platform directories. Both place their adapters in a Build subdirectory, not a Tools directory. Another interesting thing is the props file which points to platform independent versions of referenced assemblies.

Here is the layout for the MSTest adapter,

Notice that the Microsoft.VisualStudio.TestPlatform.TestFramework.dll is in the _common directory and only platform specific assemblies are in the platform directories. The test framework is .NET standard.

The props files in each platform directory then tell it which assemblies to use for each platform target.

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="12.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <ItemGroup>
    <Content Include="$(MSBuildThisFileDirectory)..\_common\Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll">
      <Link>Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll</Link>
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <Visible>False</Visible>
    </Content>
    <Content Include="$(MSBuildThisFileDirectory)..\_common\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.dll">
      <Link>Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.dll</Link>
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <Visible>False</Visible>
    </Content>
    <Content Include="$(MSBuildThisFileDirectory)..\uap10.0\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.dll">
      <Link>Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.dll</Link>
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <Visible>False</Visible>
    </Content>
  </ItemGroup>
</Project>

I am not sure what the targets files are for, fallback resource files for translations?

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="12.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <EnableMSTestV2CopyResources Condition="$(EnableMSTestV2CopyResources) == ''">true</EnableMSTestV2CopyResources>
  </PropertyGroup>

  <Target Name="GetMSTestV2CultureHierarchy">
    <!-- Only traversing 5 levels in the culture hierarchy. This is the maximum lenght for all cultures and should be sufficient to get to a culture name that maps to a resource folder we package. 
    The root culture name for all cultures is invariant whose name is ''(empty) and the parent for invariant culture is invariant itself.(https://msdn.microsoft.com/en-us/library/system.globalization.cultureinfo.parent(v=vs.110).aspx.) 
    So the below code should not break build in any case. -->
    <ItemGroup>
      <CurrentUICultureHierarchy Include="$([System.Globalization.CultureInfo]::CurrentUICulture.Name)" />
      <CurrentUICultureHierarchy Include="$([System.Globalization.CultureInfo]::CurrentUICulture.Parent.Name)"  Condition="$([System.Globalization.CultureInfo]::CurrentUICulture.Parent.Name) != ''"/>
      <CurrentUICultureHierarchy Include="$([System.Globalization.CultureInfo]::CurrentUICulture.Parent.Parent.Name)"  Condition="$([System.Globalization.CultureInfo]::CurrentUICulture.Parent.Parent.Name) != ''"/>
      <CurrentUICultureHierarchy Include="$([System.Globalization.CultureInfo]::CurrentUICulture.Parent.Parent.Parent.Name)"  Condition="$([System.Globalization.CultureInfo]::CurrentUICulture.Parent.Parent.Parent.Name) != ''"/>
      <CurrentUICultureHierarchy Include="$([System.Globalization.CultureInfo]::CurrentUICulture.Parent.Parent.Parent.Parent.Name)"  Condition="$([System.Globalization.CultureInfo]::CurrentUICulture.Parent.Parent.Parent.Parent.Name) != ''"/>
    </ItemGroup>
  </Target>

  <!-- Copy resources over to $(TargetDir) if this is a localized build. -->
  <Target Name="CopyMSTestV2Resources" BeforeTargets="PrepareForBuild" Condition="$(EnableMSTestV2CopyResources) == 'true'" DependsOnTargets="GetMSTestV2CultureHierarchy">
    <ItemGroup>
      <MSTestV2ResourceFiles Include="$(MSBuildThisFileDirectory)..\_common\%(CurrentUICultureHierarchy.Identity)\*resources.dll">
        <CultureString>%(CurrentUICultureHierarchy.Identity)</CultureString>
      </MSTestV2ResourceFiles>

      <Content Include="@(MSTestV2ResourceFiles)" Condition="@(MSTestV2ResourceFiles) != ''">
        <Link>%(MSTestV2ResourceFiles.CultureString)\%(Filename)%(Extension)</Link>
        <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
        <Visible>False</Visible>
      </Content>
    </ItemGroup>
  </Target>

  <!-- This is required because an empty resource folder is left even though the files within are cleaned up. -->
  <Target Name="CleanupMSTestV2ResourceFolders" AfterTargets="AfterClean" Condition="$(EnableMSTestV2CopyResources) == 'true'" DependsOnTargets="GetMSTestV2CultureHierarchy">
    <ItemGroup>
       <ResourceDirectories Include="$(TargetDir)%(CurrentUICultureHierarchy.Identity)" />
    </ItemGroup>
    <!-- RemoveDir does not throw if the folder does not exist. Continue on error - In any case do not fail build if this task fails(Warn and move on).-->
    <RemoveDir Directories="@(ResourceDirectories)" ContinueOnError="true"/>
  </Target>
</Project>

[rprouse]

Here is the layout of the xUnit adapter,

[galvesribeiro]

@rprouse the new SDK removes a lot of boilerplate code from the .csproj because it automatically import/infer some props/targets from the .Net SDK. Those props are targeting the correct reference for each one of the underlying multitargeted platforms.

I would suggest take that approach for the Engine and any dependency library. The test adapter can be a netstandard library. Don't need to multitarget I guess.

Regarding the console... There are 2 ways to do it AFAIK. (1) Use the test adapter SDK as xUnit guys did and (2) create a Tool which I don't think is well documented yet on .Net SDK. The tool can be used while building the project or as a standalone exe. But I would not be concerned about it. Everyone targeting netstandard (even people on net4xx) will eventually fall in dotnet CLI, so they will end up using dotnet test from now on. If people stay behind not using dotnet CLI, they can just use the old test mode/nugets/runner IMHO.

[codito]

@CharliePoole @jnm2 the dotnet test runner already looks into msbuild to find the target frameworks, accordingly it loads up the test adapters within corresponding runtime context (testhost.exe for net46, dotnet testhost.dll for netcoreapp). As long as the nunit engine and adapter are available for net46, netcoreapp1.0, test runs will just work :)

@jnm2 if a library is netstandard14;net46 and needs to be tested on UWP, NETCore and NET46; user can multitarget the test project only to uap10.0;netcoreapp1.0;net46. dotnet test takes over from there. netstandard14 build of product library is automatically tested in uap10.0 and netcoreapp1.0 runtime contexts. To test a single target framework, user can do dotnet test -f netcoreapp1.0.

Hope it clarifies the use case a bit :)

@rprouse both mstest and xunit leverage nuget's build extensibility to copy over testadapters to output directory of test project. dotnet test tries to find and load possible test adapters (*.TestAdapter.dll regex) from test project's dependency closure. In VS 2017, vstest.console.exe also tries to load test adapters from output directory (default if /TestAdapterPath command line switch is not specified).

MSTest does include an extra target to copy localized resources.

[jnm2]

@codito Thank you! I'm guessing this is a follow up from https://github.com/dotnet/sdk/issues/833#issuecomment-278151809?

If I understand correctly, the current guidance is to tell people not to target netstandard at all in the top-level test projects being executed, but rather multitarget them to specific platforms and versions like you're saying. This gives the test runner a DLL per platform and version and the test runner should be able to tell which platform and version to use by examining each DLL.

Here's a question I'm struggling with: if the NUnit framework projects take advantage of multitargeting, they'll probably consolidate to a single project targeting net20;net35;net40;net45;netstandard1.6;.NETPortable,Version=v4.5,Profile=Profile259.

What should the framework test projects should be targeting? Due to preprocessor conditions, each of these six framework targets runs slightly different code.

That means we'd want to:

• run tests on the net20 framework DLL on net20
• run tests on the net35 framework DLL on net35
• run tests on the net40 framework DLL on net40
• run tests on the net45 framework DLL on net45
• run tests on the netstandard1.6 framework DLL on netcore1.0
• run tests on the netstandard1.6 framework DLL on net47 (maybe)
• run tests on the .NETPortable,Version=v4.5,Profile=Profile259 framework DLL on net45
• run tests on the .NETPortable,Version=v4.5,Profile=Profile259 framework DLL on win8 or netcore1.0 (maybe)

In particular, the test projects need to end up running tests against net45 twice: once for the net45 framework DLL, and once for the portable framework DLL.

The simple transformation net20;net35;net40;net45;netstandard1.6;.NETPortable,Version=v4.5,Profile=Profile259 -> net20;net35;net40;net45;netcoreapp1.0;net45 doesn't really make sense anymore. What's a good way to handle this?

[codito]

.NETPortable,Version=v4.5,Profile=Profile259 is same as netstandard1.0 (see pcl vs netstandard spec).

Let's take the conflicting cases:

1. run tests on the net45 framework DLL on net45
2. run tests on the netstandard1.0 framework DLL on net45

Here dotnet tooling (nuget) will ensure 2 will never happen.

There's tool to find how nuget resolves the nearest dependency: http://nugettoolsdev.azurewebsites.net/3.5.0-beta2-1484/get-nearest-framework?project=net45&package=netstandard1.0%0D%0Anet45

In the example above, NUnit will provide netstandard1.0 only to support test execution on win8. netstandard1.6 will be always picked up for netcoreapp1.0.

[jnm2]

As far as the NUnit framework's own tests, they use project refs and not NuGet. I haven't looked but currently I don't think any tests are run on win8. I'm pretty sure we will want a way to run the portable DLL tests on net45. I guess that will mean retaining a separate framework .csproj just for the portable build, and a separate test .csproj just to reference the separate portable framework .csproj. Is that the best option in the situation?

If someday target execution platforms could be specified per target, that would allow a single .csproj for the framework and a single .csproj for the test project which could target netstandard1.6 and netstandard1.0 and run each on a couple of platforms, as well as the net* targets each on its platform.

[codito]

I guess that will mean retaining a separate framework .csproj just for the portable build...

Yes, this seems like best option to me. The other option is manually copying over the netstandard1.0 framework dll along side net45 test, before running the tests.

Theoretically, netstandard1.0 should be tested on win8 only because that's the customer use case. I understand there is a practical aspect where win8 adds a new platform to test matrix and makes developer's life bit difficult, and net45 can be used there :)

[rprouse]

I am going to use this issue to track the .NET Standard work and close all duplicates.

[rprouse]

@CharliePoole or @OsirisTerje can you tell me why the test adapter must target .NET 3.5? Visual Studio 2012 targets .NET 4.5 as do the MSTest and XUnit adapters.

I am having trouble with multi-targeting back to 3.5 and would like to switch to 4.5, but since you have a unit test for 3.5, I assume it is for a reason.

[OsirisTerje]

@rprouse You are correct. Yes, we need to keep the adapter at the lowest .net framework, and likewise the included packages. Afaic remember this is because the runner and then the adapter runs in the same domain as the code under test, so if that runs in 3.5 the adapter must do so too.

[rprouse]

@OsirisTerje if the framework level is an issue, why does the MSTest adapter target 4.6 and xUnit adapter 4.5.x?

[CharliePoole]

@rprouse Do those adapters support testing lower targeted tests under the target framework or do they just run in the higher-level framework? We always made it a requirement to suppor execution under the target framework.

[rprouse]

@CharliePoole that is why I am asking. We don't launch agents, so the adapters just run in whatever framework they start up in. I am pretty sure that is just the version of .NET CLR that the version of Visual Studio (vstest.exe) is compiled as. VS2017 adds to that the ability to launch different .NET versions, but as far as I can tell, that is limited to .NET 4.6, UWP and .NET Core.

[OsirisTerje]

The MSTest2 adapter can not run under 4.0 or lower, it requires 4.5, as NUnit can. I would assume that the same applies to XUnit. This only affects the test projects of course, but for some organisations just having to update all their test projects to 4.5 might not be viable. However, since the 2 other frameworks dont support it, perhaps we could consider a break at a certain point. It could be in version 4. Thoughts?

[rprouse]

Good news for everyone who is tracking this issue, I have my first running .NET Core tests in Visual Studio 2017,

Bad news, dotnet test isn't working yet,

λ dotnet test .\NetCoreTests\NetCoreTests.csproj
Build started, please wait...
Build completed.

Test run for C:\src\nunit\NUnitPlatformTests\NetCoreTests\bin\Debug\netcoreapp1.0\NetCoreTests.dll(.NETCoreApp,Version=v1.0)
Microsoft (R) Test Execution Command Line Tool Version 15.0.0.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
No test discoverer is registered to perform discovery of test cases. Register a test discoverer and try again.

[dmitriyse]

Good news! I managed to build this project! Great job. We are started to use this project in the production. But please publish 4.0.0-alpha1 version of nuget package somewere.

[dmitriyse]

Also I managed to create executable project also visible for VS tests UI.

You need to add this to "*.csproj"

...
  <PropertyGroup>
    <TargetFrameworks>netcoreapp1.0;netcoreapp1.1;net46</TargetFrameworks>
    <!--Required to compile-->
    <GenerateProgramFile>false</GenerateProgramFile>
    <OutputType>exe</OutputType>

Then you can add this startup:

using System;
using System.Reflection;
using NUnit.Common;
using NUnitLite;

namespace Python.EmbeddingTest
{
    public class Program
    {
        public static int Main(string[] args)
        {
            return new AutoRun(typeof(Program).GetTypeInfo().Assembly)
                .Execute(args, new ExtendedTextWrapper(Console.Out), Console.In);
        }
    }
}

For CI you can use dotnet run ... Also you can use VS Test UI.

[rprouse]

@dmitriyse I will start a PR so that this project starts building and you can track it. It isn't ready for release to NuGet.org yet, so you will need to modify your nuget.config file to add our AppVeyor nuget feeds.

Look at https://github.com/rprouse/NUnitPlatformTests/blob/master/NuGet.Config and https://github.com/rprouse/NUnitPlatformTests/blob/master/NetCoreTests/NetCoreTests.csproj if you want to get up and running now.

As for building for .NET 3.5, the issue that I am running into is Microsoft/msbuild#1333. I could get it to work by building using msbuild instead of dotnet which I may do, but I only want to do that if it is absolutely necessary. Can anyone think of a reason why we need to target .NET 3.5 in the adapter since we only support back to 2012 and VSTest.exe runs in the version of .NET that VS runs as?

[dmitriyse]

<PackageReference Include="NUnit3TestAdapter" Version="4.0.0-ci-00405-issue-297" />

Thank you for pointing.

I think .Net 3.5 and lower is most likely dead. Except for Win98 users :)

[OsirisTerje]

@rprouse I answered this in the other thread. The adapter runs in the version that the target under test is running, not in the version that vstest exe is running. If the adapter doesn't support 3.5 targets, it can't run tests that are targeting 3.5.
We need to decide if we're completely dropping support for 3.5 in the tests. MSTest and XUnit does not support this.

[rprouse]

@rprouse I answered this in the other thread. The adapter runs in the version that the target under test is running, not in the version that vstest exe is running. If the adapter doesn't support 3.5 targets, it can't run tests that are targeting 3.5.

@OsirisTerje thanks. I created a test suite targeting .NET 2.0 with the following test,

[Test]
public void TestMethod()
{
    TestContext.WriteLine("CLR: " + Environment.Version.ToString());
}

The output from the released version of the adapter is, CLR: 4.0.30319.42000, so it is running .NET 4.x when in Visual Studio. I can't see a way to change that.

Running from the vstest.console.exe command line, I can use the /Framework:framework35 option to run under .NET 3.5. Framework20 isn't supported.

I'd be curious how many people actually run their tests under .NET 3.5 using vstest, but it is possible, so we likely should support it.

[OsirisTerje]

I get this:

Same code. What are we doing different?

[rprouse]

I figured out the difference. My project targeted .NET 2.0. I guess that since /Framework:framework20 isn't supported, they just run it 4.x. When I retargeted to 3.5 I see the same output as you.

[OsirisTerje]

Good :-) Then we just need to figure out how to deal with it. If we can't support .net 3.5 when also .net standard/core is involved, then imho we should let support for 3.5 stay at NUnit3 adapter and we could let the new adapter be version 4.
Systems supporting both .net standard/core and .net 3.5 in the test projects sounds very unlikely. And, since the software under test can still be 3.5 and its test projects be 4.0++ it should be less of a problem for most others who really want to upgrade the adapter.
My 5 cents.....

[galvesribeiro]

I'm all up for @OsirisTerje suggestion. Create a new version and from that one, drop net35.

[CharliePoole]

What exactly does it mean to say "we can't support .net 3.5 when also .net standard/core is involved?" Clearly any given test assembly targets one or the other, not both.

[OsirisTerje]

Yes, but if a solution contains multiple projects, some using .net standard/core and others using framework 3.5 then it will require a testadapter that can cover both.

[CharliePoole]

Are we sure that's true? Why does the same adapter have to do both? Is the same process used for both?

[OsirisTerje]

We can of course run two different adapters for this. That is what I meant above, one for covering 3.5 targets, which would be the NUnit3 adapter series, and then the proposed NUnit4 adapter which would cover the .net standard/core projects, and anything but the 3.5 projects.
There is however an issue with tests that could possibly be covered by both adapters, and that is standard projects targeting .net 4.0 and above. There is no way the adapters can be "directed" to run only for a given project. They are all loaded in and executes over all the projects, running those they can run.
But do we really need to cover all these [fringe] cases?
If we skip support for .net 3.5 in a proposed 4 adapter, then people can still use the 3 adapter to run their stuff. And the likelihood of someone combining .net 3.5 and standard/core should be extremely small.

[rprouse]

The same adapter can support .NET 3.5 and .NET Core, it is just more work, so I wanted to make sure we needed 3.5.

• Because 3.5 is older, I need more #if blocks in the code or more compatibility classes
• Microsoft/msbuild#1333 means that I cannot build and pack the adapter using the dotnet command line, but instead need to use msbuild. This is mainly just redoing build.cake again and I have had trouble with MSBuild 14 getting pulled in instead of 15 on the CI machines. Nothing insurmountable, just more work 😄

I don't think maintaining two adapters going forward is a good idea unless we are saying the v3 adapter isn't going to see ongoing work. I think we should continue to move it forward as one project including our original plan of running the NUnit 2 tests.

I have a branch where I started the .NET 3.5 work. I will carry on with the intention of supporting 3.5.

[OsirisTerje]

Ok, this means we will have sort of the same code base, at least the files will be the same, but the #if blocks will separate things. I guess that is workable, as long as the blocks are not too big nor too many. But we will still have two releases then.