Closed igisav closed 6 years ago
@igisav For all Unity versions 5.6 and above currently we don't have a good solution, which would allow running nunit tests in Unity-generater solution without Unity running.
In the past we tried to update nunit-framework.dll from Unity one to the original one, but it caused side-effects https://youtrack.jetbrains.com/issue/RIDER-14825
I hope we finally will find a consistent solution.
What if you have Unity and Rider running and connection indicator shows "Connected" and then run tests? Does it work for you?
@van800 No. If I run test with "Connected", I see pending state of tests, then progression clock on Unity icon, and then tests converted to inconclusive. Some strange Rider exceptions popups (see below) But after some little changes in code, recompiling and switching between Rider and Unity, suddenly tests are running....I could not find a reasonable explanation.
--- EXCEPTION #1/2 [InvalidOperationException]
Message = “An attempt was made to transition a task to a final state when it had already completed.”
ExceptionPath = Root.InnerException
ClassName = System.InvalidOperationException
HResult = COR_E_INVALIDOPERATION=80131509
Source = mscorlib
StackTraceString = “
at System.Threading.Tasks.TaskCompletionSource`1[TResult].SetResult (TResult result) [0x00013] in <4343323a00204b0b9f5e7ca3dfd716be>:0
at JetBrains.ReSharper.Plugins.Unity.Rider.UnitTesting.RunViaUnityEditorStrategy+<>c__DisplayClass11_0.<RunInternal>b__1 (JetBrains.Platform.Unity.EditorPluginModel.RunResult result) [0x00000] in <7784dad26cd94e81b5185f57617e441d>:0
at JetBrains.Platform.RdFramework.Util.Signal`1[T].Fire0 (System.Action`1[T][] listeners, T value) [0x0000a] in <97f41334bc0f41e7ba293e79afec26b2>:0
”
--- Outer ---
--- EXCEPTION #2/2 [LoggerException]
Message = “An attempt was made to transition a task to a final state when it had already completed.”
ExceptionPath = Root
ClassName = JetBrains.Util.LoggerException
InnerException = “Exception #1 at Root.InnerException”
HResult = COR_E_APPLICATION=80131600
StackTraceString = “
at JetBrains.Util.LoggerBase.Log (JetBrains.Util.LoggingLevel level, System.String message, System.Exception ex) [0x00000] in <6045c331b4334e3e8f3f917b0d597942>:0
at JetBrains.Util.Logging.Log+SwitchingLog.JetBrains.Util.Logging.ILog.Log (JetBrains.Util.LoggingLevel level, System.String message, System.Exception exception) [0x00000] in <983372af6d8a4f249425ee2c68b3cd00>:0
at JetBrains.Util.Logging.LogEx.Error (JetBrains.Util.Logging.ILog this, System.Exception ex, System.String message) [0x00000] in <983372af6d8a4f249425ee2c68b3cd00>:0
at JetBrains.Platform.RdFramework.Util.Signal`1[T].Fire0 (System.Action`1[T][] listeners, T value) [0x00000] in <97f41334bc0f41e7ba293e79afec26b2>:0
at JetBrains.Platform.RdFramework.Util.Signal`1[T].Fire (T value) [0x00000] in <97f41334bc0f41e7ba293e79afec26b2>:0
at JetBrains.Platform.RdFramework.Impl.RdSignal`1+<>c__DisplayClass12_0[T].<.ctor>b__1 (JetBrains.Util.PersistentMap.UnsafeReader stream) [0x00000] in <97f41334bc0f41e7ba293e79afec26b2>:0
at JetBrains.Platform.RdFramework.Impl.MessageBroker.Execute (System.Action`1[T] handler, JetBrains.Platform.RdFramework.RdId id, System.Byte[] msg) [0x00000] in <97f41334bc0f41e7ba293e79afec26b2>:0
at JetBrains.Platform.RdFramework.Impl.MessageBroker+Subscription+<>c__DisplayClass5_0.<Invoke>b__0 () [0x00000] in <97f41334bc0f41e7ba293e79afec26b2>:0
It may be related to the way we install plugin. I would speed up improvement https://github.com/JetBrains/resharper-unity/issues/510 and then I guess the problem will disappear.
I am keeping this issue for running simple unit-tests without connected Unity.
duplicated by https://youtrack.jetbrains.com/issue/RIDER-15415
@igisav I have tried to close both applications, pressing the sync button in unity, switch back and forth many times, recompile both assemblies several times.. i can not make it work. Can you remember if you did something special?
I have not been able to make the test runner work. Tried "steps" above, as in restart, recompile, reconnect etc, still not running the test, only "pending". Both applications are running, and rider states its connected to rider.
I reproduced the problem with Unity 5.6.3.
[ERROR]14:52:40.687
Could not load file or assembly 'UnityEngine.TestRunner, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies.
at JetBrains.Rider.Unity.Editor.Ge56.UnitTesting.ModelAdviceExtension+<>c.<AdviseUnitTestLaunch>b__0_0 (JetBrains.Platform.Unity.EditorPluginModel.UnitTestLaunch launch) [0x00000] in <filename unknown>:0
at JetBrains.Platform.RdFramework.Util.Signal`1[JetBrains.Platform.Unity.EditorPluginModel.UnitTestLaunch].Fire0 (System.Action`1[] listeners, JetBrains.Platform.Unity.EditorPluginModel.UnitTestLaunch value) [0x00000] in <filename unknown>:0
@tactile-readonly what is your Unity version?
@van800 we currently use 5.6.3p4.
@tactile-readonly try to do some changes in your code (i.e. unit tests) and run test runner in Unity. But, anyway, I don't know... it's like alchemy and guessing
Fix for Unity 5.6.3 is on the way https://github.com/JetBrains/resharper-unity/pull/531
Fantastic, thank you!
Nightly build with fix (Unity 5.6.3 problem) is already available via https://www.jetbrains.com/toolbox/app/ Please try and let us know, how it goes.
So, running basic unit tests now works. However, for tests using the [Values()] attribute, it says "Test not run".
Got this exception in Unity:
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> Mono.Cecil.AssemblyResolutionException: Failed to resolve assembly: 'System.Runtime, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
at Mono.Cecil.BaseAssemblyResolver.Resolve (Mono.Cecil.AssemblyNameReference name, Mono.Cecil.ReaderParameters parameters) [0x000f7] in 1[T] parameters) [0x0002e] in <c6cbe17310154746b5cee213d0ed414a>:0 at Mono.Cecil.MetadataReader.ReadCustomAttributeSignature (Mono.Cecil.CustomAttribute attribute) [0x0003c] in <c6cbe17310154746b5cee213d0ed414a>:0 at Mono.Cecil.CustomAttribute.<Resolve>b__35_0 (Mono.Cecil.CustomAttribute attribute, Mono.Cecil.MetadataReader reader) [0x00000] in <c6cbe17310154746b5cee213d0ed414a>:0 at Mono.Cecil.ModuleDefinition.Read[TItem,TRet] (TItem item, System.Func
3[T1,T2,TResult] read) [0x00029] in 1[T] assemblyPaths, APIUpdater.Framework.Configuration.IConfigurationProvider provider) [0x00017] in <79eb5983cd354eedb9ed613dbacce245>:0 at AssemblyUpdater.Application.Program.ConfigurationProviderFrom (System.Collections.Generic.IEnumerable
1[T] configurationAssemblies) [0x00020] in <2e2e6e53129e4ba1874d2ad0eec22c59>:0
at AssemblyUpdater.Application.Program.CheckForObsoleteAPIUsage (AssemblyUpdater.Application.CommandLineSpec config) [0x0007a] in <2e2e6e53129e4ba1874d2ad0eec22c59>:0
at AssemblyUpdater.Application.Program.Main (System.String[] args) [0x00057] in <2e2e6e53129e4ba1874d2ad0eec22c59>:0
UnityEditor.AssetDatabase:Refresh()
JetBrains.Rider.Unity.Editor.AdditionalPluginsInstaller:UpdateSelf(String)
JetBrains.Platform.RdFramework.Util.Signal1:Fire0(Action
1[], String)
JetBrains.Platform.RdFramework.Util.Signal1:Fire(String) JetBrains.Platform.RdFramework.Util.RProperty
1:set_Value(String)
JetBrains.Platform.RdFramework.Impl.<>cDisplayClass21_0:
@trondtactile Please provide some example, which works for you in Unity.
using NUnit.Framework;
using Assert = UnityEngine.Assertions.Assert;
public class ExampleTest {
[Test]
public void WorkingTest() {
Assert.IsTrue(true);
}
[Test]
public void NonRunningTest([Values(true, false)] bool value) {
Assert.IsTrue(value);
}
}
@trondtactile Thanks! Please expected fix in a couple of days. https://github.com/JetBrains/resharper-unity/commit/29c3dd0c693e7e721d8bdd0c3570c33bd35738ec
awesome, thanks!
Hello!
Any status update on this update?
I am using the most recent JetBrains Rider nightly build and am still unable to run NUnit unit tests from Rider itself. They forever display "Pending" until I cancel them, at which time they display "Inconclusive".
Thank you and be well! - S.
also have "pending" problem on the latest nightly building JetBrains Rider 2018.1 EAP Build #RD-181.4952.17, built on May 2, 2018 Unity 2018.1.0f2
I also noticed that you still get "unconclusive" if you have the attribute [TestCase()] on the test method. (similar to the [values] attributes)
TestCase, ValueSource and other dynamic tests are in the works https://github.com/JetBrains/resharper-unity/pull/551
Dynamic tests support was added in the latest available nightly build. https://rider-support.jetbrains.com/hc/en-us/articles/360000009804-How-to-get-Rider-Nightly-build-
I am closing this issue in favor of #530
Just upgraded to the newest nightly build. Now i can not run non-dynamic tests even. Rider tries to build the application whenever i try to run tests, which fails like so:
Microsoft.Common.CurrentVersion.targets(1601, 5): [MSB4036] The "GetReferenceNearestTargetFrameworkTask" task was not found. Check the following: 1.) The name of the task in the project file is the same as the name of the task class. 2.) The task class is "public" and implements the Microsoft.Build.Framework.ITask interface. 3.) The task is correctly declared with <UsingTask> in the project file, or in the *.tasks files located in the "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin" directory.
I am sorry for inconvenience. You may try to change Launch - Build Policy to "Never" Or if it doesn't help then fix MSBuild installation (https://developercommunity.visualstudio.com/content/problem/137779/the-getreferencenearesttargetframeworktask-task-wa.html)
I will try to come up with better solution later today
@trondtactile As far as I can see, if you have selected to run tests via UnityEditor and you have connection to UnityEditor, build is not fired. If the problem reproduces for you, let me see the logs from Rider->Help->Show log in Files, collect all log files please.
@trondtactile any update on your last issue? Also please check the nie nightly build with multiple fixes.
@van800 I updated to nightly build #RD-181.4952.153 and still have tests in pending or inconclusive state... Reinstalling plugin, switch between Unity and Rider or recompiling doesn't help. How we can provide more information for you?
yeah, tried the latest as well. Disabled building when running tests. Now they are just stuck at pending
instead.
Please provide me with your OS, Unity version. And I will check them. If you can spend some time to help, here is what you can do:
This issue will be exclusively about running tests via Unity. And #530 will be about running tests without Unity via NUnit.
I am sorry, I found out that build #RD-181.4952.153 haven't got all the fixes. Please wait for the next one.
Related to RIDER-15809 Can't run unit tests (pong-tdd)
New nightly build is out. Please try. I checked Unity versions 5.6.3f1, 2017.4.2f2, 2018.1.0f2, 2018.2.0b4 on Windows.
I tested on Mac the new nightly build and it works. (RD-181.4952.191, macOS 10.12.6, Unity 2018.1.0b12)
Yesterday I tried on Windows older Version with Unity 2018.1.0.b5 and got following error in logs:
RiderPlugin[ERROR]22:42:53.472 Exception while launching Unity Editor tests.
Constructor on type 'UnityEditor.TestTools.TestRunner.EditModeLauncher' not found.
at System.RuntimeType.CreateInstanceImpl (System.Reflection.BindingFlags bindingAttr, System.Reflection.Binder binder, System.Object[] args, System.Globalization.CultureInfo culture, System.Object[] activationAttributes, System.Threading.StackCrawlMark& stackMark) [0x00213] in <dcaae11f4f674b16b268094ee0049a09>:0
at System.Activator.CreateInstance (System.Type type, System.Reflection.BindingFlags bindingAttr, System.Reflection.Binder binder, System.Object[] args, System.Globalization.CultureInfo culture, System.Object[] activationAttributes) [0x000a1] in <dcaae11f4f674b16b268094ee0049a09>:0
at System.Activator.CreateInstance (System.Type type, System.Reflection.BindingFlags bindingAttr, System.Reflection.Binder binder, System.Object[] args, System.Globalization.CultureInfo culture) [0x00000] in <dcaae11f4f674b16b268094ee0049a09>:0
at JetBrains.Rider.Unity.Editor.UnitTesting.UnityEditorTestLauncher.TryLaunchUnitTests () [0x00337] in <ff0e2bf33edf46c0a2aa82d3d0cf880a>:0
RECV[TRACE]22:51:47.283 endpoint `UnityEditorPlugin.EditorPluginModel.getUnityEditorState`::(7454109928747411040), taskId=1000465, request = JetBrains.Platform.RdFramework.RdVoid
SEND[TRACE]22:51:47.283 endpoint `UnityEditorPlugin.EditorPluginModel.getUnityEditorState`::(7454109928747411040), response = Success :: Idle
RECV[TRACE]22:51:48.241 endpoint `UnityEditorPlugin.EditorPluginModel.getUnityEditorState`::(7454109928747411040), taskId=1000467, request = JetBrains.Platform.RdFramework.RdVoid
SEND[TRACE]22:51:48.241 endpoint `UnityEditorPlugin.EditorPluginModel.getUnityEditorState`::(7454109928747411040), response = Success :: Idle
But maybe you found already the problem cause and fix it. Thanx a lot.
@igisav yes, I think I fixed that one. Thanks for the feedback.
I can confirm this now works for me. Thanks! 👍
I can't get it to work with 2019.1 Rider and Unity 2018.4. It says inconclusive test not run. I can run the same tests from Unity's test runner window and they work as expected. It's annoying I can't from rider and we have far more options for running tests than in Unity. The tests are also very simple and not dependant on any MonoBehaviour or unity specific code. It's testing code of static classes.
Any idea what is causing this?
@lordlycastle If tests are not Unity ones, without access to GameObjects, you can switch TestRunner to use Standalone nUnit.
@lordlycastle Please check in Unity that your tests are Editor (not Player) ones. I tried 2018.4 with EditorTests and it worked just fine.
@van800 I tried that "Standalone NUnit Launcher". I get the following error:
System.Security.SecurityException : ECall methods must be packaged into a system module.
TearDown : System.Security.SecurityException : ECall methods must be packaged into a system module.
at UnityEngine.DebugLogHandler.LogFormat(LogType logType, Object context, String format, Object[] args)
at UnityEngine.Logger.Log(LogType logType, Object message)
at *** MY TEST FILE.cs ***
--TearDown
at UnityEngine.DebugLogHandler.LogFormat(LogType logType, Object context, String format, Object[] args)
at UnityEngine.Logger.Log(LogType logType, Object message)
at *** MY TEST FILE.cs ***
However, we will add some unity tests later and will require to use Unity Editor mode.
@van800 I don't understand what you mean by "Editor not Player".
@lordlycastle you may check in Unity test runner
ECall methods must be packaged into a system module
means tests are calling Unity native API-s, which can't be called in standalone nunit runner, only in real Unity environment.
@van800 Yes it in Edit mode and tests show up there and run fine.
You're right, I was using a Debug.Log
in the tear down. Is there a preprocessor flag that can be use to switch between Stanalone Nunit and Unity Editor
@lordlycastle This would work:
try
{
Debug.Log("test");
}
catch
{
Console.WriteLine("test");
}
@lordlycastle I am curious to see what is causing the issue, when running in UnityEditor mode from Rider.
Could you send me some sample project to Ivan.Shakhov@jetbrains.com? Other option is to Enable Trace logs in Unity Preferences Rider, reproduce the issue and send me all logs from the folder %Temp%\Unity3dRider Thanks!
@van800 I've sent you a test project with the issue at hand. Thanks for the help.
We had a chat with @lordlycastle about his case with TestCaseSource.
public struct MultiplyTestCase
{
public int result;
public (int, int) input;
}
public int Multiply(int x, int y) => x * y;
private static readonly List<MultiplyTestCase> _testCases =
new List<MultiplyTestCase>
{
new MultiplyTestCase {result = 4, input = (2, 2),},
// Fail test case.
new MultiplyTestCase {result = 5, input = (2, 3),},
// Success test case.
new MultiplyTestCase {result = 9, input = (3, 3),}
};
[Test, TestCaseSource(nameof(_testCases))]
public void TestMultiply(MultiplyTestCase testCase)
{
Assert.AreEqual(testCase.result, Multiply(testCase.input.Item1, testCase.input.Item2));
}
In Unity it produces 3 tests with similar names: But for Rider it displays only one test. Workaround is to override a ToString() in MultiplyTestCase:
public override string ToString()
{
return input + "=" + result;
}
In general I believe we can't do much with it from Rider side, it requires a fix in Unity.
After updgrading to Rider 2018.1 and running unit tests without starting Unity on my Mac give me 'inconclusive' result. If I start Unity, then tests stay in pending mode without any response....
a part of stacktrace: