dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
14.61k stars 4.56k forks source link

[NativeAOT] Object synchronization method was called from an unsynchronized block of code. #104340

Open jkotas opened 1 week ago

jkotas commented 1 week ago

Intermittent Object synchronization method was called from an unsynchronized block of code. exceptions.

Known Issue Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "Object synchronization method was called from an unsynchronized block of code.",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: :mag_right: Result validation: :warning: Validation could not be done without an Azure DevOps build URL on the issue. Please add it to the "Build: :mag_right:" line. Validation performed at: 7/3/2024 2:40:39 AM UTC

Report

Build Definition Step Name Console log Pull Request
741963 dotnet/runtime Build Tests Log dotnet/runtime#104904
740232 dotnet/runtime Build Tests Log dotnet/runtime#103861
740112 dotnet/runtime Build Tests Log dotnet/runtime#104857
736278 dotnet/runtime Build Tests Log dotnet/runtime#104687
736103 dotnet/runtime Build Tests Log dotnet/runtime#104681
735305 dotnet/runtime Build Tests Log dotnet/runtime#103735
735014 dotnet/runtime Build Tests Log dotnet/runtime#104603
733925 dotnet/runtime Build Tests Log dotnet/runtime#104445
732512 dotnet/runtime Build Tests Log dotnet/runtime#104534
730825 dotnet/runtime Build Tests Log dotnet/runtime#104440
729981 dotnet/runtime Build Tests Log dotnet/runtime#101580
729766 dotnet/runtime Build Tests Log dotnet/runtime#104379
728548 dotnet/runtime Build Tests Log dotnet/runtime#104332

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
1 7 13
dotnet-policy-service[bot] commented 1 week ago

Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas See info in area-owners.md if you want to be subscribed.

jkotas commented 1 week ago

Hit in #104202. Log https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-104202-merge-fe38091c4b294deca8/System.Threading.Channels.Tests/1/console.5ad789fe.log?helixlogtype=result :

[FAIL] System.Threading.Channels.Tests.UnboundedPrioritizedChannelTests.ManyProducerConsumer_ConcurrentReadWrite_Success(numReaders: 1, numWriters: 10)
System.AggregateException : One or more errors occurred. (Object synchronization method was called from an unsynchronized block of code.)
---- System.Threading.SynchronizationLockException : Object synchronization method was called from an unsynchronized block of code.
   at System.Threading.Tasks.Task.WaitAllCore(ReadOnlySpan`1, Int32, CancellationToken) + 0x304
   at System.Threading.Tasks.Task.WaitAll(Task[]) + 0x54
   at System.Threading.Channels.Tests.ChannelTestBase.ManyProducerConsumer_ConcurrentReadWrite_Success(Int32 numReaders, Int32 numWriters) + 0x16c
   at System.Threading.Channels!<BaseAddress>+0xaa3254
jkotas commented 1 week ago

Hit in #104332. Log https://dev.azure.com/dnceng-public/public/_build/results?buildId=728548&view=logs&j=eb3d96e8-a17f-53b1-ecea-be64723f4b14&t=d8e271dd-8628-5dad-a374-176afa32935b&l=673 :

    Generating native code
    Generating native code
    Unhandled exception: System.AggregateException: One or more errors occurred. (Object synchronization method was called from an unsynchronized block of code.)
     ---> System.Threading.SynchronizationLockException: Object synchronization method was called from an unsynchronized block of code.
       at System.Threading.ObjectHeader.Release(Object) + 0x14a
       at System.Threading.Monitor.Exit(Object) + 0x17
       at System.Resources.ResourceReader.GetValueForNameIndex(Int32) + 0x253
       at System.Resources.ResourceReader.ResourceEnumerator.get_Value() + 0xdd
       at ILCompiler.SubstitutionProvider.AssemblyFeatureInfo..ctor(EcmaModule, Logger, IReadOnlyDictionary`2, BodyAndFieldSubstitutions) + 0x5eb
       at ILCompiler.SubstitutionProvider.FeatureSwitchHashtable.CreateValueFromKey(EcmaModule) + 0x5a
       at Internal.TypeSystem.LockFreeReaderHashtable`2.CreateValueAndEnsureValueIsInTable(TKey) + 0x3b
       at Internal.TypeSystem.LockFreeReaderHashtable`2.GetOrCreateValue(TKey) + 0x62
       at ILCompiler.SubstitutionProvider.GetSubstitution(MethodDesc) + 0xaa
       at ILCompiler.SubstitutedILProvider.GetMethodIL(MethodDesc) + 0x67
       at ILCompiler.Compilation.CombinedILProvider.GetMethodIL(MethodDesc) + 0x4f
       at ILCompiler.Compilation.ILCache.CreateValueFromKey(MethodDesc) + 0x6a
MichalStrehovsky commented 6 days ago

I think the failure in ILC.exe started happening after #103574 (this is ILC-compiled ILC, using the LKG ILC), so that narrows it down to like two weeks worth of commits -_-

ILC.exe failures don't have crashdumps because of infra limitations and neither does the libraries test failure (again because of infra design).

MichalStrehovsky commented 3 days ago

I spent some time digging through pipeline logs and trying to repro locally. We've not seen this in libraries testing in the past 10+ days, only when running ILC. We picked up a new ILC on Wednesday and it hasn't reproed since. I was not able to get a local repro either (I was already running with the updated ILC though).

So fingers crossed that whatever this was, it was already fixed and we just need to wait a couple more days to get enough confidence to close this issue.