Cannot run unit tests in a project.json style project that targets multiple runtimes

jeremydmiller commented 8 years ago

If you want, you can pull down the StructureMap solution at http://github.com/structuremap/structuremap to see what I mean here. When I try to run any test - or all of the tests - with TD.Net I get:

StructureMap.Testing.dll doesn't exist.
Please ensure that the project has been compiled.

When you look at the file system, dotnet is compiling it to: bin/Net451, but it drops all of its dependencies into a platform specific folder underneath that, "win7-x64" in my case.

gmartinezsan commented 8 years ago

Hitting this too, with the following error: System.Exception: Couldn't find type with name 'DataTypeTests' at MutantDesign.Xml.Documentation.MemberInfoUtilities.FindMemberInfo(Assembly assembly, String cref) at TestDriven.Framework.TestRunnerTarget.GetTestRunnerTarget(String assemblyFile, String cref) at TestDriven.NetCore.TestRunnerFactory.Create(ITestListener testListener, String assemblyFile, String cref, TestRunnerTarget& target) at TestDriven.NetCore.Program.run(String assemblyFile, String cref) at TestDriven.NetCore.Program.Remote.Run(String assemblyFile, String cref)

jcansdale commented 8 years ago

@jeremydmiller, @gmartinezsan,

I'm just about to release a new version (release notes always seem to take forever 😉).

Could you try again with this build? TestDriven.NET-4.0.3425_Extraterrestrial_Beta2.zip

The notes will look something like this: https://github.com/jcansdale/TestDriven.Net-Issues/blob/master/ReleaseNotes.md

This version appears to work with @jeremydmiller's StructureMap.Testing project.

@gmartinezsan, I don't know if it will make a difference for your project. If it doesn't, do you have some source I could try it with?

gmartinezsan commented 8 years ago

@jcansdale I'll give it a try and let you know. Thanks a lot!

jcansdale commented 8 years ago

@gmartinezsan,

I've fixed an issue that is possibly related to this: https://github.com/jcansdale/TestDriven.Net-Issues/issues/63

Could you give this updated build a spin and let me know how you get on? TestDriven.NET-4.0.3428_Extraterrestrial_Beta2.zip

gmartinezsan commented 8 years ago

hi @jcansdale Still getting an exception with the extraterrestrial version when trying to run one of our tests. Any suggestions ?

System.Exception: Couldn't find type with name 'DataTypeTests' at MutantDesign.Xml.Documentation.MemberInfoUtilities.FindMemberInfo(Assembly assembly, String cref) at TestDriven.Framework.TestRunnerTarget.GetTestRunnerTarget(String assemblyFile, String cref) at TestDriven.NetCore.TestRunnerFactory.Create(ITestListener testListener, String assemblyFile, String cref, String testRunnerName, TestRunnerTarget& target) at TestDriven.NetCore.Program.run(String assemblyFile, String cref, String testRunnerName) at TestDriven.NetCore.Program.Main(String[] args)

jcansdale commented 8 years ago

This is a strange one! Does it work with other test classes in this assembly? Is the class public?

Could you strip down the project so there is only this class and any sensitive code has been removed? Having a repro would really help.

gmartinezsan commented 8 years ago

I saw what the problem is. Is something very simple.

My solution supports .net451 and netcoreapp 1.0. And the issue I think is related to the selected framework in the code editor. There is a point where the framework is not valid, I guess, because I cannot select any in the code editor of Visual Studio.

If I have selected in the code editor the 4.5.1 framework and I run a different one from the Test Driven context menu I got the error I mentioned. If in both cases I use the same framework, everything work as expected. So I would say this build works fine with my solution. I will be using it more and if I found more details I'll let you know. Thank you a lot for your quick response!

jcansdale commented 8 years ago

I'm glad you found what the issue is!

Any chance you could share the source file with DataTypeTests in? (or a trimmed down summary) I'd be interested to see exactly what's going on.

gmartinezsan commented 8 years ago

Sure.. actually I think you can reproduce it with any project that targets multiple frameworks. Anyways I will create one and see what are the steps to reproduce it and will share it here. So you can check what the problem is.

gmartinezsan commented 8 years ago

@jcansdale I'm back at checking the unit tests suite of my project, Now I'm getting this error when trying to debug a unit test from any of the classes. Although running them works fine. My project targets .net 4.5.1 and core.

Is this a known issue?

I'm using latest beta for testdriven.net

System.MethodAccessException: Attempt to access method Xunit.FactAttribute..ctor() in violation of security transparency rules failed. at System.RuntimeMethodHandle.CheckLinktimeDemands(IRuntimeMethodInfo method, RuntimeModule module, Boolean isDecoratedTargetSecurityTransparent) at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeModule decoratedModule, Int32 decoratedMetadataToken, Int32 pcaCount, RuntimeType attributeFilterType, Boolean mustBeInheritable, IList derivedAttributes, Boolean isDecoratedTargetSecurityTransparent) at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeMethodInfo method, RuntimeType caType, Boolean inherit) at System.Attribute.GetCustomAttributes(MemberInfo element, Boolean inherit) at System.Reflection.CustomAttributeExtensions.GetCustomAttributes(MemberInfo element) at TestDriven.NetCore.TestRunnerFactory.hasCustomAttributes(MemberInfo member) at TestDriven.NetCore.TestRunnerFactory.useAdHoc(MemberInfo member) at TestDriven.NetCore.TestRunnerFactory.Create(ITestListener testListener, String assemblyFile, String cref, String testRunnerName, TestRunnerTarget& target) at TestDriven.NetCore.Program.run(String assemblyFile, String cref, String testRunnerName) at TestDriven.NetCore.Program.Main(String[] args)

jeremydmiller commented 8 years ago

@jcansdale I think this one is fixed, or at least it is for me in later builds.

jcansdale commented 8 years ago

@gmartinezsan,

System.MethodAccessException: Attempt to access method Xunit.FactAttribute..ctor()
     in violation of security transparency rules failed.

This looks looks like a weird one, especially since it only happens when debugging! I'm able to debug a simple xUnit test just fine. I wonder what might be different about your tests? Does this happen for all of your projects or are there any where debugging works?

@jeremydmiller,

I'm pleased to hear your issue seems to be sorted! 😄

gmartinezsan commented 8 years ago

@jcansdale Thank you for your attention Jamie, the solution I'm working on is a class library. As mentioned here http://stackoverflow.com/questions/36118567/how-to-switch-between-target-frameworks-for-net-core-projects-in-visual-studio looks like there are some prerequisites to be switching between frameworks inside the code editor of Visual Studio. This seems to have still an issue. I thought that having the project of the unit tests as startup project would solved it but it doesn't change anything. If the project targets only one framework everything works fine. Please let me know if you need further details.

jcansdale commented 7 years ago

Switching views seems to work for me on console and class library projects. It is however only a view and Visual Studio always executes using the first framework defined in the "frameworks" section. I can't even detect which view is selected (if I wanted to make this work differently in TestDriven.Net).

Could you upload a simple repro here? I haven't been able to trigger the exception mentioned.

Thanks, Jamie.

gmartinezsan commented 7 years ago

@jcansdale Finally took the time to create this solution and was able to have the repro steps.

https://github.com/gmartinezsan/ConsoleAppForTrace Feel free to let me know if you need further details.

Steps to reproduce:

Open the solution in VS 2015
Go to the Mytestclass.cs file
Add a breakpoint inside the FirstTest method
Select the 4.5.1 framework in the combo box of the code editor.
Right clic on the context menu in the FirstTest method (same as when starting the debugger).
Select "Test With" menu and then select .NET Framework
On the output window you will see the following error:

------ Test started: Assembly: ConsoleTests.dll ------

System.Exception: Couldn't find declaring type with name 'ConsoleTests.Mytestclass' at MutantDesign.Xml.Documentation.MemberInfoUtilities.FindMemberInfo(Assembly assembly, String cref) at TestDriven.Framework.TestRunnerTarget.GetTestRunnerTarget(String assemblyFile, String cref) at TestDriven.NetCore.TestRunnerFactory.Create(ITestListener testListener, String assemblyFile, String cref, String testRunnerName, TestRunnerTarget& target) at TestDriven.NetCore.Program.run(String assemblyFile, String cref, String testRunnerName) at TestDriven.NetCore.Program.Remote.Run(String assemblyFile, String cref, String testRunnerName)

If the net core app is selected instead then it does work.

When using the debugger

I only can debug the first framework of my project.json file. For example if the first framework is .net451 then that's the only framework I can debug even if I have selected the .net core app in the combo of the code editor. It is debugging correctly the .net451.

Hope this helps!

kzu commented 7 years ago

FWIW, selecting the target framework is implemented now in VS2017 Preview (15.3), and it basically sets an ActiveDebugFramework MSBuild property in the .user file alongside the .csproj.

May (or may not ;)) be useful to @jcansdale ;)

jcansdale commented 7 years ago

@kzu Thanks for the heads up. I'm messing with the 15.3 Preview at the moment...

jcansdale commented 7 years ago

@kzu I've created a multi-target project, but can't find how to select the target framework. Could you give me a clue? 😄

I'm using 15.3 and my .csproj includes: <TargetFrameworks>net462;net46;netstandard2.0</TargetFrameworks>

kzu commented 7 years ago

It has to be an SDK-style project, and you'd see the dropdown in the Run button:

jcansdale commented 7 years ago

Got it, thanks!

I'd never have thought to look in *.csproj.user.

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="15.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <ShowAllFiles>false</ShowAllFiles>
    <ActiveDebugFramework>net46</ActiveDebugFramework>
  </PropertyGroup>
</Project>

I guess that could also be used as a handy place to inject stuff into a build. I'm like to be able to right-click on a VSSDK project and have it install/execute in a different version of VS. Maybe *.csproj.user. could be the gateway. 😄

jcansdale / TestDriven.Net-Issues

Cannot run unit tests in a project.json style project that targets multiple runtimes #31