Failed to execute OSPlatform.get_Linux() on macOS 12 using dotnet 3.1

DenisRumyantsev commented 3 years ago

Description

Hi, I am currently working on the issue which Azure DevOps customers face trying to configure ADO agent on Mac devices running on macOS 12 Monterey Beta 3 platform.

Here are steps to reproduce the issue:

Exact platform version where ADO agent is tried to configure: macOS 12 Monterey Beta 3 (build version: 21A5284e)
Exact ADO agent which is tried to configure: v2.189.0 (earlier versions were tried with the same result)
Exact command which is run to configure ADO agent (within bin directory of an agent): ./Agent.Listener configure --unattended --replace --url "https://<organization_name>.visualstudio.com" --pool "<pool_name>" --agent "<agent_name>" --work "_work" --runasservice --auth "PAT" --token "<actual_token>"

The error message which customers get after running the command mentioned above:

Process terminated. Infinite recursion during resource lookup within System.Private.CoreLib.  This may be a bug in System.Private.CoreLib, or potentially in certain extensibility points such as assembly resolve events or CultureInfo names.  Resource name: Arg_AccessViolationException
   at System.Environment.FailFast(System.String)
   at System.SR.InternalGetResourceString(System.String)
   at System.SR.GetResourceString(System.String, System.String)
   at System.AccessViolationException..ctor()
   at System.Reflection.CustomAttribute.FilterCustomAttributeRecord(System.Reflection.MetadataToken, System.Reflection.MetadataImport ByRef, System.Reflection.RuntimeModule, System.Reflection.MetadataToken, System.RuntimeType, Boolean, ListBuilder`1<System.Object> ByRef, System.RuntimeType ByRef, System.IRuntimeMethodInfo ByRef, Boolean ByRef, Boolean ByRef)
   at System.Reflection.CustomAttribute.AddCustomAttributes(ListBuilder`1<System.Object> ByRef, System.Reflection.RuntimeModule, Int32, System.RuntimeType, Boolean, ListBuilder`1<System.Object>)
   at System.Reflection.CustomAttribute.GetCustomAttributes(System.Reflection.RuntimeModule, Int32, Int32, System.RuntimeType)
   at System.Reflection.CustomAttribute.GetCustomAttributes(System.Reflection.RuntimeAssembly, System.RuntimeType)
   at System.Reflection.RuntimeAssembly.GetCustomAttributes(System.Type, Boolean)
   at System.Attribute.GetCustomAttributes(System.Reflection.Assembly, System.Type, Boolean)
   at System.Attribute.GetCustomAttribute(System.Reflection.Assembly, System.Type, Boolean)
   at System.Reflection.CustomAttributeExtensions.GetCustomAttribute[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.Reflection.Assembly)
   at System.Resources.ManifestBasedResourceGroveler.GetNeutralResourcesLanguage(System.Reflection.Assembly, System.Resources.UltimateResourceFallbackLocation ByRef)
   at System.Resources.ResourceManager.CommonAssemblyInit()
   at System.Resources.ResourceManager..ctor(System.Type)
   at System.SR.get_ResourceManager()
   at System.SR.InternalGetResourceString(System.String)
   at System.SR.GetResourceString(System.String, System.String)
   at System.AccessViolationException..ctor()
   at System.Runtime.InteropServices.OSPlatform..ctor(System.String)
   at System.Runtime.InteropServices.OSPlatform..cctor()
   at System.Runtime.InteropServices.OSPlatform.get_Linux()
   at Agent.Sdk.PlatformUtil.get_HostOS()
   at Agent.Sdk.PlatformUtil.get_RunningOnWindows()
   at Microsoft.VisualStudio.Services.Agent.Util.StringUtil..cctor()
   at Microsoft.VisualStudio.Services.Agent.Util.StringUtil.ConvertToBoolean(System.String, Boolean)
   at Agent.Sdk.Knob.KnobValue.AsBoolean()
   at Agent.Sdk.PlatformUtil.get_UseLegacyHttpHandler()
   at Microsoft.VisualStudio.Services.Agent.Listener.Program.Main(System.String[])
Abort trap: 6

The main idea is that ADO agent uses dotnet 3.1 and is run on macOS 12 Monterey Beta 3 platform, and the error occurs here when trying to execute RuntimeInformation.IsOSPlatform(OSPlatform.Linux), or more detailed:

System.Runtime.InteropServices.RuntimeInformation.IsOSPlatform(System.Runtime.InteropServices.OSPlatform.Linux)

so please see at the string at System.Runtime.InteropServices.OSPlatform.get_Linux() and above it in the error logs.

dotnet-issue-labeler[bot] commented 3 years ago

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

danmoseley commented 3 years ago

There is no way that OSPlatform..ctor(System.String) should AV. The process state is probably already corrupted. That is then causing a second AV in FilterCustomAttributeRecord.

some ideas

Is it possible to narrow down the repro case? Eg., partially roll back your changes, or simplify.
If you can attach windbg just before the failure, load SOS and run !VerifyHeap https://docs.microsoft.com/en-us/dotnet/framework/tools/sos-dll-sos-debugging-extension
You might be able to set up ProcDump to catch the AV and save a dump fie. https://docs.microsoft.com/en-us/sysinternals/downloads/procdump

@AaronRobinsonMSFT any other thoughts? Nominally this type is yours, although the connection to your code (and indeed this type) is tenuous

AaronRobinsonMSFT commented 3 years ago

@danmoseley I agree with your analysis here – at least conceptually. My one question would be regarding CrossGen, Linker, or any post-build processing impact. I could imagine a case where IL was removed in some scenarios and this causes a cascade of failures for Linux on a non-Linux platform (i.e., macOS). Other than the comment on procdump – which is Windows only I believe – I think your ideas are the most appropriate.

danmoseley commented 3 years ago

Oops yes not Windows. I'd be interested to know if there's a tool similar to procdump on Mac or Linux.

AaronRobinsonMSFT commented 3 years ago

@danmoseley I believe the equivalent on macOS would be dtruss. On other unix platforms I would typically start with strace.

danmoseley commented 3 years ago

that's procmon ? I meant procdump. Essentially, a tool to capture dumps based on unhandled exceptions, events, timers etc.

AaronRobinsonMSFT commented 3 years ago

@danmoseley Doh. Sorry about that. I honestly don't know if there is a ProcDump equivalent. Perhaps @janvorli knows of something.

janvorli commented 3 years ago

On Linux, there is a gcore tool, but all it can do is to take a dump of a running process when started. gdb can be used to catch AV and take a dump.

On macOS, you have two options:

You can run a process under lldb or attach lldb and then take a dump (process save-core LLDB command). It looks like the error above happens early during startup, so running it under lldb would be easy.
You can enable core dumps and let the OS create the dump on the abort. It seems that the evidence on where the AV happened would likely still be there.

To enable dumps, it is needed to:

set the access right of the /cores folder so that the user running the code has write access (or add the user to the wheel group)
run ulimit -c unlimited from the shell where the application will be launched from (or from the same script before launching the application)
I had an experience with latest macOS versions that it works only with bash shell and not the now default zsh for some reason.

DenisRumyantsev commented 3 years ago

@AaronRobinsonMSFT I run a process under lldb, and after the error occurred, made a core dump using process save-core lldb command. I sent you a link to the core dump in Microsoft Teams, could you please take a look?

AaronRobinsonMSFT commented 3 years ago

@DenisRumyantsev I did take a look and have responded on Teams. I am quoting the response below:

I loaded the shared coredump file. Unfortunately it wasn't done with the .NET createdump binary. We found some issues regardless and recorded them at https://github.com/dotnet/symstore/issues/289. I will set aside some time today to debug this but I'm not sure what is going on here and the stack doesn't really make a lot of sense. Also, this is a preview version of the macOS so it is entirely possible there is a bug here that we are sensitive to. It is also possible this will require a servicing fix for .NET Core 3.1 to service so the problem here could be an early warning of that. The short answer is there are limited details here and I have few abilities to actually root cause this issue with the current environment. I assume this is an early pass on a preview OS and not a production blocking issue though. Please correct me if I am wrong. Using createdump might actually help here - https://docs.microsoft.com/dotnet/core/diagnostics/debug-linux-dumps.

danmoseley commented 3 years ago

@DenisRumyantsev can you confirm that/which of these work?

macOS 12 using .NET 5+
macOS <12 using .NET 3.1

DenisRumyantsev commented 3 years ago

@danmoseley I am currently building and testing agents with different dotnet runtime versions and checking if there is this issue on macOS 12 for them. I will provide you the results when I'm done with this.

DenisRumyantsev commented 3 years ago

@danmoseley @AaronRobinsonMSFT I have upgraded the dotnet version to the latest compatible one, with which our agent successfully passes the build for macOS (dotnet runtime 3.1.15 version instead of 3.1.0 which we are currently using), and the error is no longer reproduced on macOS 12 Beta. It looks like this issue has already been fixed and we need to upgrade the agent to a higher version of dotnet. So I'm closing this issue.

danmoseley commented 3 years ago

@DenisRumyantsev thanks for the update. great.

BTW, 3.1.15 is out of date. The latest version is 3.1.18 https://dotnet.microsoft.com/download/dotnet/3.1

dotnet / runtime

Failed to execute OSPlatform.get_Linux() on macOS 12 using dotnet 3.1 #56725

Description