MarcoRossignoli
commented
1 month ago @jkotas do you know who owns System.Console? We see an issue only on native aot mode related to some writeline.
Open martincostello opened 1 month ago
MarcoRossignoli
commented
1 month ago @jkotas do you know who owns System.Console? We see an issue only on native aot mode related to some writeline.
MarcoRossignoli
commented
1 month ago @Evangelink @nohwnd do you recall if some code inside MSTest is doing some wild lock on Console.Out?
MarcoRossignoli
commented
1 month ago @martincostello ResilienceHandler.cs is your code?
martincostello
commented
1 month ago @MarcoRossignoli I believe it's this: https://github.com/dotnet/extensions/blob/main/src/Libraries/Microsoft.Extensions.Http.Resilience/Resilience/ResilienceHandler.cs
jkotas
commented
1 month ago @jkotas do you know who owns System.Console?
You can find dotnet/rutime area owners in https://github.com/dotnet/runtime/blob/main/docs/area-owners.md
However, this does not look like a System.Console bug to me. The stacktraces do not explain why there is a deadlock. It is more likely a bug in native AOT runtime . Have you tried to run latest .NET 9 Preview? It may be a bug that was fixed in .NET 9.
martincostello
commented
1 day ago While trying to work out why my Lambda function isn't working when deployed to AWS Lambda with RC.1, I managed to by-accident generate a crash dump on both Linux and macOS. The Linux one showed a deadlock on two threads trying to do a Console.WriteLine():
I can share a link to the crash dump file over email. This was in normal xunit, rather than anything to do with AoT, but the commonality appears to be access to the console.
jkotas
commented
1 day ago This is likely duplicate of https://github.com/dotnet/runtime/issues/104340 that will be fixed in .NET 9 RC2.
Describe the bug
Today I've come back to the underlying issue that lead to me opening #3097 to try and work out what's going wrong. I think I've now got a fairly good idea of where the issue is (but not why) and have worked around the issue.
From trial and error in https://github.com/martincostello/alexa-london-travel/pull/1368, I managed to get hold of a dump on Linux under WSL and got the stacks for each thread: https://github.com/martincostello/alexa-london-travel/pull/1368#issuecomment-2271045338
The interesting stacks were the following:
It looks like there's a deadlock between trying to write to the console via
ILoggerin the code under test and the MSTest platform trying to change the colour of the console at the end of the test run before reporting the test summary.It may be that this is an issue in .NET rather than VSTest, but I figured I'd start here in the first instance as I only seem to get this problem using the native AoT test support.
Some interesting points in the code are listed below:
ConsoleDisplayService.DisplayAfterSessionEndRunAsync()sets the console colour just before writing the test summary.SystemConsole.SetForegroundColor()to setConsole.ForegroundColor.Console.ForegroundColorcalls through toConsolePal.RefreshColors().RefreshColors()locks onConsole.Out.SyncTextWriter.Write()- this uses[MethodImpl(MethodImplOptions.Synchronized)]on the methods, so I think that's effectively also locking onConsole.Out.The fact that only the Linux implementation of the setter of
Console.ForegroundColorappears to do any locking would also help explain why I only see this on Linux.I've worked around this issue by reducing the amount of console output from the tests. With the workaround applied, the tests happily keep passing if I run them continuously in a loop (until the job timeout limit of 20 minutes is reached).
However, what I'm at a loss to explain is why there's a deadlock - just that there appears to be one on usage of the console.
Steps To Reproduce
build.ps1Expected behavior
The tests pass and exit successfully.
Actual behavior
The tests deadlock.