xunit / visualstudio.xunit

VSTest runner for xUnit.net (for Visual Studio Test Explorer and dotnet test)
https://xunit.net/
Other
144 stars 81 forks source link

Catastrophic error during deserialization #382

Closed CaringDev closed 1 year ago

CaringDev commented 1 year ago

While trying to bring forward a successor for xBehave, I most likely run into a misunderstanding of how to extend xUnit. After getting rid of {Platform} specific Execution projects I get the following Test output. The tests are shown in VS Test Explorer but cannot be run (Rider, R# in VS and dotnet test work as expected). I'm aware of Why doesn't xUnit.net support netstandard? but thought this is related to runners / running and not assemblies, that need to be loaded.

Please find here a MRE.

========== Starting test discovery ==========
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.5.0.1+5ebf84cd75 (64-bit .NET 7.0.10)
[xUnit.net 00:00:00.05]   Discovering: TestProject1
[xUnit.net 00:00:00.08]   Discovered:  TestProject1
========== Test discovery finished: 2 Tests found in 365.2 ms ==========
Executing all tests in solution
========== Starting test run ==========
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.5.0.1+5ebf84cd75 (64-bit .NET 7.0.10)
[xUnit.net 00:00:00.03] TestProject: Catastrophic error during deserialization: System.ArgumentException: Could not load type 'LambdaTale.Execution.ScenarioOutlineTestCase, LambdaTale.{Platform}' from serialization value 'LambdaTale.Execution.ScenarioOutlineTestCase, LambdaTale.{Platform}:VGVzdE1ldGhvZDpYdW5pdC5TZGsuVGVzdE1ldGhvZCwgeHVuaXQuZXhlY3V0aW9uLntQbGF0Zm9ybX06VFdWMGFHOWtUbUZ0WlRwVGVYTjBaVzB1VTNSeWFXNW5PbEZ1VmpCU2JVWndZa2hOUFFwVVpYTjBRMnhoYzNNNldIVnVhWFF1VTJSckxsUmxjM1JEYkdGemN5d2dlSFZ1YVhRdVpYaGxZM1YwYVc5dUxudFFiR0YwWm05eWJYMDZWa2RXZW1SRlRuWmlSM2hzV1ROU2NHSXlORFpYU0ZaMVlWaFJkVlV5VW5KTWJGSnNZek5TUkdJeWVITmFWMDR3WVZjNWRVeERRalJrVnpWd1pFTTFiR1ZIVm1wa1dGSndZakkwZFdVeFFuTlpXRkp0WWpOS2RHWlVjRk5TTW5nMldUQmtOR0ZIVmxaT1YyaHBWakZWTWxaVVRuTmxiVkpJVm01U1RXSkZOSGRaTWpGelpGWndObU5HWkZOTlZtOHlWMnRXVDFFeVJuUlRXR3hzVTBVMWFGWnFRVEJrTVd4WFdYcFdhMVpYZUVsV01qVmhZVzFHVmxOc1dsaGlSMmhRVkZWYVYxTldUblZYYldocFZteHdlRmRyVmxOU2JWSlhWbGhzYUZOR2NISldhazV2WTJ4YWNWTlVWbXhXTUZsNlZVWlJkMU14V2toV2JuQnJVbFZhTmxsNlNsZGtSbXgwWlVSV1VHSkhaM2haYlRGelRVVjRjMVJ0ZEdobFZGWldWMnhvVDAxR1JsbFVibkJoVm5wR2NGbHJhSEpqTUd4SllVUkdhV0pYZDNkVVJ6RlhUa1p3V0ZScVJtdFNNbmd5V1cxck1VNHhWa2hsUjJoclVqRndNbGt5TUhoUFZUbHpVbXhzVldKdVFtaFdibkJIWTBac2NtRklUbFpXYlhoYVZsY3dOVlZIU2taT1JFWmFaV3MxVkZsclpFdFdSVFZYVlcxMFZGSllRak5YVnpCNFlXczFjMWR1U21GU2JWSnpXV3hhUzFOR1ZsaE9XRTVvVmxSR01WVlhOVTlYUmxwR1RsWk9ZVkpzY0ROVk1HUlhVMVpTVm1SR1NrNWhiWGQ0VmpJd2VHTXlTbk5UYmxKcFUwVTFhRlpzVWxkaU1XeHpXa1phVDJKR1ZqWlphMmgzWVRGSmVGZHFRbFZTTTAweFdWWmtSMlJHV25SalIzUlNWMFZLV1ZZeWVHOVRNbFowVW14b1VGZEdXbEZaYTFVd1RWWnNObFJzVG1sU01IQlZWR3hhVTJFeFRrWmpTR1JhWWxSR2NWUnRlRk5XTVZKVlVXMTBUbE5IYURSV1ZscHZWakZTYzFKc1ZrNVNSM1JNVm14ak1XTkhUbGxXYlhoVVZsWkZNbFpVVG5ObGJWSklWbTVTVFdKRk5IZFpNakZ6WkZad05tTkZPVmRTVkVJMlZrWldVMVJ0VmtaTldFWlNWMFpLVUZSWGRGZE9WbEp6VkZoa1RsSnRkekpWTWpFMFZGWlpkMk5IT1ZWV00wSkRXa1ZWZUZaWFNraE5WbkJYVWxad2VGWkhOWGRaVjBwSFlraEdVbGRIWkV4VmEyUlhZVzFLU0ZKdWJGcFhSa3AzV1dwSk1WRnRUWHBVYlhocFZqQndlbHBXVlRGaFIwcFlWbFJhVmsweWVEWmFSV1JYWkVWNGNrOVhiR2hpVmxweFdrVkdkMUpXY0ZoVWJrNWFWMFZ3YjFwRlpITmtiVXB6VldwV2FsSXhXbEJYVm1ONFlrVTVjMVJxVm1wTk1VcHpXV3hOTVZWR2JIUmpSM2hhVFRGRk9VTnJUbk5aV0U1NlVWaE9lbHBYTVdsaVNHeFBXVmN4YkU5c1RqVmpNMUpzWWxNMVZHUklTbkJpYldNMlZtdGtWMlZ0VWtkUmJteHBUVzVDYzFkVVRsSmpNR3hIVjIxNGFtSnJOWGRaYWtrd1QxVXhWRTVJWkUxaGEwWXhWRlZPTTFveFJYcFdiazVyVTBaYU5WZHNVWGhrVm5CWlZtcENhbUpWV25wVVJVNURWVmRTV0ZOdVRtaFdNRFZOVjJ4b2MxWlhTWGxrUjNocFlXcEdNVnBHWkRSamQzQkVZa2RHZW1NeFVqVmpSMVpQV1ZjeGJFOXNUalZqTTFKc1lsTTFWR1JJU25CaWJXTTJWbXRrVjJWdFVrZFJibXhwVFc1Q2MxZFVUbEpsUlhoelZHMDVhVTB4V25wWGExcHJaRzFPZEdONk1EMD0KVGVzdE1ldGhvZEFyZ3VtZW50czpTeXN0ZW0uT2JqZWN0CkRlZmF1bHRNZXRob2REaXNwbGF5OlN5c3RlbS5TdHJpbmc6UTJ4aGMzTkJibVJOWlhSb2IyUT0KRGVmYXVsdE1ldGhvZERpc3BsYXlPcHRpb25zOlN5c3RlbS5TdHJpbmc6VG05dVpRPT0KVGltZW91dDpTeXN0ZW0uSW50MzI6MA==' (Parameter 'serializedValue')
   at Xunit.Sdk.SerializationHelper.Deserialize[T](String serializedValue) in /_/src/common/SerializationHelper.cs:line 35
   at Xunit.Sdk.TestFrameworkExecutor`1.Deserialize(String value) in /_/src/xunit.execution/Sdk/Frameworks/TestFrameworkExecutor.cs:line 66
   at Xunit.Sdk.XunitTestFrameworkExecutor.Deserialize(String value) in /_/src/xunit.execution/Sdk/Frameworks/XunitTestFrameworkExecutor.cs:line 88
   at Xunit.DefaultTestCaseBulkDeserializer.<BulkDeserialize>b__2_0(String serialization) in /_/src/xunit.runner.utility/Descriptor/DefaultTestCaseBulkDeserializer.cs:line 22
   at System.Linq.Utilities.<>c__DisplayClass2_0`3.<CombineSelectors>b__0(TSource x)
   at System.Linq.Enumerable.SelectListIterator`2.ToList()
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at Xunit.DefaultTestCaseBulkDeserializer.BulkDeserialize(List`1 serializations) in /_/src/xunit.runner.utility/Descriptor/DefaultTestCaseBulkDeserializer.cs:line 22
   at Xunit.Xunit2.BulkDeserialize(List`1 serializations) in /_/src/xunit.runner.utility/Frameworks/v2/Xunit2.cs:line 74
   at Xunit.XunitFrontController.BulkDeserialize(List`1 serializations) in /_/src/xunit.runner.utility/Frameworks/XunitFrontController.cs:line 122
   at Xunit.Runner.VisualStudio.VsTestRunner.RunTestsInAssembly(IRunContext runContext, IFrameworkHandle frameworkHandle, LoggerHelper logger, TestPlatformContext testPlatformContext, RunSettings runSettings, IMessageSinkWithTypes reporterMessageHandler, AssemblyRunInfo runInfo) in /_/src/xunit.runner.visualstudio/VsTestRunner.cs:line 524
[xUnit.net 00:00:00.04]   Starting:    TestProject
[xUnit.net 00:00:00.05]   Finished:    TestProject
========== Test run finished: 0 Tests (0 Passed, 0 Failed, 0 Skipped) run in 62 ms ==========

I cannot find any mention of the {Platform} string. Searching for the Catastrophic error during deserialization lead me to trying to find an ominous VS Test Explorer Extension cache dir but this seems to be gone.

Would you have any hints?

CaringDev commented 1 year ago

Fixed by https://github.com/bbvch/LambdaTale/pull/14/files If anyone has an explanation... I'm still interested :-)

bradwilson commented 1 year ago

This definitely seems like a bug, as we shouldn't be doing the {Platform} replacement unless you ask for it. I'm glad you've found a workaround, though I worry that you'll be broken if someone tries to use your library on .NET Framework (which will try to substitute desktop instead of dotnet for the {Platform} replacement token).

bradwilson commented 1 year ago

I'm aware of Why doesn't xUnit.net support netstandard? but thought this is related to runners / running and not assemblies, that need to be loaded.

Yes, that's true, though the library you're trying to extend has two different versions (.NET Framework vs. everything else) because there is functionality that cannot be implementation with just .NET Standard 1.1, which is what this library multi-targets to (net452 and netstandard1.1). If you search the source of xunit.execution for #if NETFRAMEWORK you can see where those differences are.

bradwilson commented 1 year ago

Ah! This is why:

https://github.com/bbvch/LambdaTale/blob/c9ac4e9a01f88d8fa438124f31dae5764ebb7572/src/LambdaTale.Execution/Properties/AssemblyInfo.cs#L3

You should remove this, as that's the flag that tells us to do the {Platform} replacement. You should also undo the changes in https://github.com/bbvch/LambdaTale/pull/14 at the same time.

bradwilson commented 1 year ago

The XML comments describe what it does: https://github.com/xunit/xunit/blob/c6073bb042a4c081e0f1c1300d75ab44d0c41aaf/src/xunit.core/Sdk/PlatformSpecificAssemblyAttribute.cs#L5-L32

/// <summary>
/// Marks an assembly as a platform specific assembly for use with xUnit.net. Type references from
/// such assemblies are allowed to use a special suffix ("My.Assembly.{Platform}"), which will
/// automatically be translated into the correct platform-specific name ("My.Assembly.desktop",
/// "My.Assembly.win8", etc.). This affects both extensibility points which require specifying
/// a string-based type name and assembly, as well as serialization.
///
/// In v2.1 and later, the supported platform target names include:
///
///   "desktop" (for desktop and PCL tests),
///   "dotnet" (everything else).
///
/// In v2.0, the following names were also supported:
/// 
///   "iOS-Universal" (for Xamarin test projects targeting iOS),
///   "MonoAndroid" (for Xamarin MonoAndroid tests),
///   "MonoTouch" (for Xamarin MonoTouch tests),
///   "universal" (for Windows Phone 8.1 and Windows 8.1 tests),
///   "win8" (for Windows 8 tests),
///   "wp8" (for Windows Phone 8 Silverlight tests).
///
/// For backward compatibility reasons, the v2.1 runners will support tests linked against
/// the v2.0 execution libraries.
///
/// Note that file names may be case sensitive (when running on platforms with case sensitive
/// file systems like Linux), so ensure that your assembly file name casing is consistent, and
/// that you use the suffixes here with the exact case shown.
/// </summary>
CaringDev commented 1 year ago

Ah, as expected: my bad πŸ™ˆ @bradwilson thank you very much for the close look and detailed explanation and even instructions πŸ™ I'll try that tomorrow πŸ‘