search

microsoft / vstest-docs

Documentation for the Visual Studio Test Platform.
Other
214 stars 149 forks source link

vstest.console.exe fails with 'Unhandled Exception: System.ArgumentException: Illegal characters in path.' on rerun #268

Open ZeddzZ opened 2 years ago

ZeddzZ commented 2 years ago

After running tests using vstest.console.exe in ADO pipeline with some failures rerun throws an exception. After short investigation I found that the reason can be with failed tests, tagged with NUnit Test case attribute, because their final name is in format MyTestName("value1", "value2"). After such test failed it is added to rerun string and on next vstest.console.exe call fails, because braces are counted as special characters and can't be used as a part of filter, even if escaped.

Here is the command created on rerun:

vstest.console.exe /TestCaseFilter:"FullyQualifiedName=TestScenarios.TestClass1.FileReturnWholePageTest|FullyQualifiedName=TestScenarios.TestClass3.ReviewSideMenuTest(\"option1\"\,\"option2\, option3\")|FullyQualifiedName=TestScenarios.TestClass2.DataEntryOrderingTest|FullyQualifiedName=TestScenarios.TestClass1.FIleReturnEsignatureTest|FullyQualifiedName=TestScenarios.TestClass3.ReviewErrorSectionTest(\"option1\"\,\"option2\, option3\")|FullyQualifiedName=TestScenarios.TestClass3.ReviewWholePageTest(\"option1\"\,\"option2\, option3\")|FullyQualifiedName=TestScenarios.TestClass4.ProjectGlobalHeaderTest|FullyQualifiedName=TestScenarios.TestClass1.FileReturnSideMenuTest|FullyQualifiedName=TestScenarios.TestClass3.ReviewWholePageTestt(\"option1\"\,\"option2\, option3\")|FullyQualifiedName=TestScenarios.TestClass3.ReviewSideMenuTest(\"option4\"\,\"option5\, option6\")|FullyQualifiedName=TestScenarios.TestClass1.FileReturnErrorSectionTest|FullyQualifiedName=TestScenarios.TestClass2.DatEntryExpandedSideSectionTest|FullyQualifiedName=TestScenarios.TestClass2.DatEntryInterviewSubSectionTest|FullyQualifiedName=TestScenarios.TestClass1.FileReturnReportLinksTest|FullyQualifiedName=TestScenarios.TestClass2.DataEntryDownloadImportRlSlipsTest|FullyQualifiedName=TestScenarios.TestClass3.ReviewReportSectionTestt(\"option1\"\,\"option2\, option3\")|FullyQualifiedName=TestScenarios.TestClass5.ProjectPropertiesWholePageTest|FullyQualifiedName=TestScenarios.TestClass2.DatEntryCollapsedSideSectionTest|FullyQualifiedName=TestScenarios.TestClass2.DataEntrySearchFieldTest|FullyQualifiedName=TestScenarios.TestClass1.FileReturnReportLinksIncorrectProjectTest|FullyQualifiedName=TestScenarios.TestClass3.ReviewErrorSectionTest\"option4\"\,\"option5\, option6\")|FullyQualifiedName=TestScenarios.TestClass1.FileReturnReportSectionTest|FullyQualifiedName=TestScenarios.TestClass2.DatEntryNotInterviewSubSectionTest|FullyQualifiedName=TestScenarios.TestClass2.DatEntryWholePageTest|FullyQualifiedName=TestScenarios.TestClass6.T1SettingsHeaderHierarchyTest" "E:\AP06\DAB06_01_work\64\s\xxx\TestScenarios.dll" /Settings:"E:\AP06\DAB06_01_work_temp\ct3fvllxq0q.tmp.runsettings" /Logger:"trx" /TestAdapterPath:"E:\AP06\DAB06_01_work\64\s"

Whole error thrown:

[error] at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.ArgumentProcessorFactory.<>c__DisplayClass20_0.b__0()

[error] at System.Lazy`1.CreateValue()

[error] at System.Lazy`1.LazyInitValue()

[error] at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.ArgumentProcessorFactory.<>c__DisplayClass20_0.b__0()

[error] at System.Lazy`1.CreateValue()

[error] at System.Lazy`1.LazyInitValue()

[error] at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.ArgumentProcessorFactory.<>c__DisplayClass20_0.b__0()

[error] at System.Lazy`1.CreateValue()

[error] at System.Lazy`1.LazyInitValue()

[error] at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.ArgumentProcessorFactory.<>c__DisplayClass20_0.b__0()

[error] at System.Lazy`1.CreateValue()

[error] at System.Lazy`1.LazyInitValue()

[error] at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.ArgumentProcessorFactory.<>c__DisplayClass20_0.b__0()

[error] at System.Lazy`1.CreateValue()

[error] at System.Lazy`1.LazyInitValue()

[error] at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.ArgumentProcessorFactory.<>c__DisplayClass20_0.b__0()

[error] at System.Lazy`1.CreateValue()

[error] at System.Lazy`1.LazyInitValue()

[error] at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.ArgumentProcessorFactory.<>c__DisplayClass20_0.b__0()

[error] at System.Lazy`1.CreateValue()

[error] at System.Lazy`1.LazyInitValue()

[error] at Microsoft.VisualStudio.TestPlatform.CommandLine.Executor.GetArgumentProcessors(String[] args, List`1& processors)

[error] at Microsoft.VisualStudio.TestPlatform.CommandLine.Executor.Execute(String[] args)

[error] at Microsoft.VisualStudio.TestPlatform.CommandLine.Program.Main(String[] args)

Here is ADO YAML code

steps: - task: VSTest@2 displayName: 'VsTest - testAssemblies' inputs: testAssemblyVer2: | \$(BuildConfiguration)\$(TestDll) !\obj** platform: '$(BuildPlatform)' configuration: '$(BuildConfiguration)' rerunFailedTests: true rerunFailedThreshold: 50 rerunMaxAttempts: 2 continueOnError: true