search

googleapis / google-api-dotnet-client

Google APIs Client Library for .NET
https://developers.google.com/api-client-library/dotnet
Apache License 2.0
1.34k stars 525 forks source link

Grpc.Core.RpcException + Crashes when using Opentelemetry Stackdriver Exporter #1699

Closed nmbro closed 3 years ago

nmbro commented 3 years ago

I'm unsure if this is even the right place to post this, feel free to send me somewhere else. It's not reliably reproducible, but managed to crash 2 different staging instances.

Runtime: GCP image: windows-server-2019-dc-core-v20200609 Windows Server Core 2019 LTSC netframework application built for .net 472 running on net48

Versions:

Exception Message:

Grpc.Core.RpcException: 'Status(StatusCode="Unknown", Detail="Stream removed", DebugException="Grpc.Core.Internal.CoreErrorDetailException: {"created":"@1605690661.501000000","description":"Error received from peer ipv4:108.177.127.95:443","file":"T:\src\github\grpc\workspace_csharp_ext_windows_x64\src\core\lib\surface\call.cc","file_line":1062,"grpc_message":"Stream removed","grpc_status":2}")'

Stacktraces:

OS Thread Id: 0x1e0 (11)
        Child SP               IP Call Site
0000000693cfe458 00007ff8ae6302b4 [HelperMethodFrame: 0000000693cfe458] 
0000000693cfe540 00007ff8a1e08e70 System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
0000000693cfe570 00007ff8a14a9aae System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
0000000693cfe5a0 00007ff8442b5884 Grpc.Core.Internal.AsyncCall`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].UnaryCall(System.__Canon)
0000000693cfe7b0 00007ff8442b3e28 Grpc.Core.DefaultCallInvoker.BlockingUnaryCall[[System.__Canon, mscorlib],[System.__Canon, mscorlib]](Grpc.Core.Method`2<System.__Canon,System.__Canon>, System.String, Grpc.Core.CallOptions, System.__Canon)
0000000693cfe938 00007ff8a27c1f7c [StubHelperFrame: 0000000693cfe938] 
0000000693cfe990 00007ff8442b3108 Grpc.Core.Interceptors.InterceptingCallInvoker.<BlockingUnaryCall>b__3_0[[System.__Canon, mscorlib],[System.__Canon, mscorlib]](System.__Canon, Grpc.Core.Interceptors.ClientInterceptorContext`2<System.__Canon,System.__Canon>)
0000000693cfea90 00007ff8442b2cae Grpc.Core.ClientBase+ClientBaseConfiguration+ClientBaseConfigurationInterceptor.BlockingUnaryCall[[System.__Canon, mscorlib],[System.__Canon, mscorlib]](System.__Canon, Grpc.Core.Interceptors.ClientInterceptorContext`2<System.__Canon,System.__Canon>, BlockingUnaryCallContinuation`2<System.__Canon,System.__Canon>)
0000000693cfebd8 00007ff8a27c1f7c [StubHelperFrame: 0000000693cfebd8] 
0000000693cfec30 00007ff8442b2be9 Grpc.Core.Interceptors.InterceptingCallInvoker.BlockingUnaryCall[[System.__Canon, mscorlib],[System.__Canon, mscorlib]](Grpc.Core.Method`2<System.__Canon,System.__Canon>, System.String, Grpc.Core.CallOptions, System.__Canon)
0000000693cfedd8 00007ff8a27c1f7c [StubHelperFrame: 0000000693cfedd8] 
0000000693cfee30 00007ff8442b2a0a Google.Cloud.Trace.V2.TraceService+TraceServiceClient.BatchWriteSpans(Google.Cloud.Trace.V2.BatchWriteSpansRequest, Grpc.Core.CallOptions)
0000000693cfee80 00007ff8442a7ad9 Google.Api.Gax.Grpc.ApiCall+GrpcCallAdapter`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].CallSync(System.__Canon, Google.Api.Gax.Grpc.CallSettings)
0000000693cfef40 00007ff8442a775f Google.Api.Gax.Grpc.ApiCallRetryExtensions+<>c__DisplayClass1_0`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].<WithRetry>b__0(System.__Canon, Google.Api.Gax.Grpc.CallSettings)
0000000693cfeff0 00007ff843f56c8f OpenTelemetry.Contrib.Exporter.Stackdriver.StackdriverTraceExporter.Export(OpenTelemetry.Batch`1<System.Diagnostics.Activity> ByRef)
0000000693cff050 00007ff843799622 OpenTelemetry.Trace.BatchExportActivityProcessor.ExporterProc()
0000000693cff0c0 00007ff8a143df12 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
0000000693cff190 00007ff8a143dd95 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
0000000693cff1c0 00007ff8a143dd65 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
0000000693cff210 00007ff8a14e3e85 System.Threading.ThreadHelper.ThreadStart()
0000000693cff460 00007ff8a27c6953 [GCFrame: 0000000693cff460] 
0000000693cff7c0 00007ff8a27c6953 [DebuggerU2MCatchHandlerFrame: 0000000693cff7c0] 
0:011> * ================================== FAILING MANAGED STACK END================================================
0:011> * ================================== FAILING NATIVE STACK ================================================
0:011> kn
  *** Stack trace for last set context - .thread/.cxr resets it
 # Child-SP          RetAddr           Call Site
00 00000006`93cfe190 00007ff8`a293d605 KERNELBASE!RaiseException+0x69
01 00000006`93cfe270 00007ff8`a293e114 clr!RaiseTheExceptionInternalOnly+0x31f
02 00000006`93cfe390 00007ff8`a1e08e70 clr!IL_Throw+0x114
03 00000006`93cfe540 00007ff8`a14a9aae mscorlib_ni!System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()$##60053EB+0x20
04 00000006`93cfe570 00007ff8`442b5884 mscorlib_ni!System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)$##6005D24+0x3e
05 00000006`93cfe5a0 00007ff8`442b3e28 0x00007ff8`442b5884
06 00000006`93cfe7b0 00007ff8`a27c1f7c 0x00007ff8`442b3e28
07 00000006`93cfe8e0 00007ff8`442b3108 clr!InstantiatingMethodStubWorker+0xbc
08 00000006`93cfe990 00007ff8`442b2cae 0x00007ff8`442b3108
09 00000006`93cfea90 00007ff8`a27c1f7c 0x00007ff8`442b2cae
0a 00000006`93cfeb80 00007ff8`442b2be9 clr!InstantiatingMethodStubWorker+0xbc
0b 00000006`93cfec30 00007ff8`a27c1f7c 0x00007ff8`442b2be9
0c 00000006`93cfed80 00007ff8`442b2a0a clr!InstantiatingMethodStubWorker+0xbc
0d 00000006`93cfee30 00007ff8`442a7ad9 0x00007ff8`442b2a0a
0e 00000006`93cfee80 00007ff8`442a775f 0x00007ff8`442a7ad9
0f 00000006`93cfef40 00007ff8`43f56c8f 0x00007ff8`442a775f
10 00000006`93cfeff0 00007ff8`43799622 0x00007ff8`43f56c8f
11 00000006`93cff050 00007ff8`a143df12 0x00007ff8`43799622
12 00000006`93cff0c0 00007ff8`a143dd95 mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)$##6003AEF+0x172
13 00000006`93cff190 00007ff8`a143dd65 mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)$##6003AEE+0x15
14 00000006`93cff1c0 00007ff8`a14e3e85 mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)$##6003AED+0x55
15 00000006`93cff210 00007ff8`a27c6953 mscorlib_ni!System.Threading.ThreadHelper.ThreadStart()$##6003C00+0x55
16 00000006`93cff250 00007ff8`a27c6858 clr!CallDescrWorkerInternal+0x83
17 00000006`93cff290 00007ff8`a27c7118 clr!CallDescrWorkerWithHandler+0x4e
18 00000006`93cff2d0 00007ff8`a28cec8f clr!MethodDescCallSite::CallTargetWorker+0x102
19 00000006`93cff3d0 00007ff8`a27c7d10 clr!ThreadNative::KickOffThread_Worker+0xffffffff`fffff02f
1a 00000006`93cff630 00007ff8`a27c7c83 clr!ManagedThreadBase_DispatchInner+0x40
1b 00000006`93cff670 00007ff8`a27c7bc2 clr!ManagedThreadBase_DispatchMiddle+0x6c
1c 00000006`93cff770 00007ff8`a27c7da7 clr!ManagedThreadBase_DispatchOuter+0x4c
1d 00000006`93cff7e0 00007ff8`a28ceb79 clr!ManagedThreadBase_FullTransitionWithAD+0x2f
1e 00000006`93cff840 00007ff8`a27cb5f5 clr!ThreadNative::KickOffThread+0xe6
1f 00000006`93cff920 00007ff8`ac1b7974 clr!Thread::intermediateThreadProc+0x8b
20 00000006`93cffbe0 00007ff8`ae5fa271 kernel32!BaseThreadInitThunk+0x14
21 00000006`93cffc10 00000000`00000000 ntdll!RtlUserThreadStart+0x21
0:011> * ================================== FAILING NATIVE STACK END================================================
jskeet commented 3 years ago

I would expect this to be more appropriate in https://github.com/open-telemetry/opentelemetry-dotnet - the exporter should potentially be catching this exception.

I suspect the team there would appreciate it if you could post a minimal example that demonstrates the problem though - using as few dependencies as possible, for example.

jskeet commented 3 years ago

Sorry, https://github.com/open-telemetry/opentelemetry-dotnet-contrib is the repo for the OpenTelemetry.Contrib.Exporter.Stackdriver package.

nmbro commented 3 years ago

I noticed while trying to write up some tests that my IDE intelli-sense does not warn me that BatchWriteSpans can throw

jskeet commented 3 years ago

Closing this, as it's now reported at https://github.com/open-telemetry/opentelemetry-dotnet-contrib/issues/53