microsoft / vstest

Visual Studio Test Platform is the runner and engine that powers test explorer and vstest.console.
MIT License
870 stars 316 forks source link

When using the dotnet cli to run Appium tests on .net9 preview5, we are now seeing exceptions. #5112

Closed PureWeen closed 1 week ago

PureWeen commented 1 week ago

Description

It looks like the way appium routes messaging is triggering vstest to now report an error when running dotnet test

image image

It's a bit odd, because this should just be debug info that appium is spitting out about the server setup. It's not really an error that's happening. I'm still reviewing through appium to see why appium outputs this information to the error output stream.

this only started with preview5 of .net 9

Steps to reproduce

I can run these tests from VSCODE and I don't receive any exceptions

Expected behavior

It should finish running with no errors

Actual behavior

it looks like it now reports the messages from appium as errors. I'm still unclear why Appium is using the error output to display server messages.

nohwnd commented 1 week ago

as we discussed interanally, we probably need to add an option to suppress errors that match a given pattern, because this will be source of problems for others as well. We are already converting some xunit output to info, so the error count is not doubled for every failing test.