[Test Failure] RemoteExecutionException: Half-way through waiting for remote process

[rzikm]

https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-heads-main-80bd4bb3d2574f7a99/System.Net.Requests.Tests/3/console.aef87555.log?helixlogtype=result)

Many tests failing, example failure

   System.Net.Tests.WebRequestTest.SendGetRequest_WithGlobalCachePolicy_AddCacheHeaders(requestCacheLevel: Reload, expectedHeaders: ["Pragma: no-cache", "Cache-Control: no-cache"]) [FAIL]
      Microsoft.DotNet.RemoteExecutor.RemoteExecutionException : Half-way through waiting for remote process.
...
      Timed out at 6/18/2024 2:07:21 PM after 60000ms waiting for remote process.

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=711383 Build error leg or test failing: Pull request:

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": ["Microsoft.DotNet.RemoteExecutor.RemoteExecutionException : Half-way through waiting for remote process."],
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=711383 Error message validated: [Microsoft.DotNet.RemoteExecutor.RemoteExecutionException : Half-way through waiting for remote process.] Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 7/3/2024 2:05:57 PM UTC

Report

Build	Definition	Test	Pull Request
743438	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.PfxIterationCountTests_X509Certificate2.Import_IterationCountLimitExceeded_ThrowsInAllottedTime	dotnet/runtime#104759
742788	dotnet/runtime	System.Net.Sockets.Tests.TelemetryTest.EventSource_SocketConnectsRemote_LogsConnectStartStop
742311	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	dotnet/runtime#104835
733117	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.PfxIterationCountTests_X509Certificate2.Import_IterationCountLimitExceeded_ThrowsInAllottedTime	dotnet/runtime#103922
732960	dotnet/runtime	System.Diagnostics.DiagnosticSource.Tests.WorkItemExecution	dotnet/runtime#103991
730974	dotnet/runtime	System.Tests.PosixSignalRegistrationTests.ExternalConsoleManipulation_RegistrationRemoved_UnregisterSucceeds
730330	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.PfxIterationCountTests_X509Certificate2.Import_IterationCountLimitExceeded_ThrowsInAllottedTime	dotnet/runtime#101221
729920	dotnet/runtime	System.Tests.PosixSignalRegistrationTests.ExternalConsoleManipulation_RegistrationRemoved_UnregisterSucceeds
711383	dotnet/runtime	System.Net.Tests.HttpWebRequestTest_Sync.SendHttpRequest_BindIPEndPoint_Success

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	3	9

[dotnet-policy-service[bot]]

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service[bot]]

Tagging subscribers to this area: @dotnet/ncl See info in area-owners.md if you want to be subscribed.

[v-wenyuxu]

Failed in: runtime-coreclr libraries-jitstress 20240708.1

Failed tests:

net9.0-windows-Release-arm64-jitminopts-Windows.11.Arm64.Open
    - Microsoft.Extensions.Hosting.WindowsServiceLifetimeTests.ExceptionOnStartIsPropagated

Error message:

 Microsoft.DotNet.RemoteExecutor.RemoteExecutionException : Half-way through waiting for remote process.
Memory load: 36

Image Name                     PID Services                                    
========================= ======== ============================================
svchost.exe                    924 BrokerInfrastructure, DcomLaunch, Power,    
                                   SystemEventsBroker                          
svchost.exe                    476 RpcEptMapper, RpcSs                         
svchost.exe                    872 LSM                                         
svchost.exe                   1044 TermService                                 
svchost.exe                   1060 CryptSvc                                    
svchost.exe                   1108 lmhosts                                     
svchost.exe                   1260 AppIDSvc                                    
svchost.exe                   1296 NcbService                                  
svchost.exe                   1332 nsi                                         
svchost.exe                   1416 netprofm                                    
svchost.exe                   1460 CoreMessagingRegistrar                      
svchost.exe                   1500 Schedule                                    
svchost.exe                   1512 TimeBrokerSvc                               
svchost.exe                   1544 UmRdpService                                
svchost.exe                   1596 ProfSvc                                     
svchost.exe                   1840 LanmanWorkstation                           
svchost.exe                   1848 UserManager                                 
svchost.exe                   1872 DispBrokerDesktopSvc                        
svchost.exe                   1920 vmicheartbeat                               
svchost.exe                   1940 Dnscache                                    
svchost.exe                   1952 vmickvpexchange                             
svchost.exe                   1996 vmicshutdown                                
svchost.exe                   2028 vmictimesync                                
svchost.exe                   2096 EventLog                                    
svchost.exe                   2204 SysMain                                     
svchost.exe                   2212 EventSystem                                 
svchost.exe                   2232 SessionEnv                                  
svchost.exe                   2248 Themes                                      
svchost.exe                   2412 CertPropSvc                                 
svchost.exe                   2428 SENS                                        
svchost.exe                   2460 Dhcp                                        
svchost.exe                   2556 AudioEndpointBuilder                        
svchost.exe                   2596 FontCache                                   
svchost.exe                   2656 Audiosrv                                    
svchost.exe                   2720 WinHttpAutoProxySvc                         
svchost.exe                   2728 TextInputManagementService                  
svchost.exe                   2848 DusmSvc                                     
svchost.exe                   2864 Wcmsvc                                      
svchost.exe                   2972 ShellHWDetection                            
svchost.exe                   1132 BFE, mpssvc                                 
svchost.exe                   3152 DiagTrack                                   
svchost.exe                   3160 DPS                                         
svchost.exe                   3176 iphlpsvc                                    
svchost.exe                   3236 LanmanServer                                
svchost.exe                   3268 StateRepository                   

Stack trace:

   at Microsoft.DotNet.RemoteExecutor.RemoteInvokeHandle.Dispose(Boolean disposing) in /_/src/Microsoft.DotNet.RemoteExecutor/src/RemoteInvokeHandle.cs:line 225
   at Microsoft.DotNet.RemoteExecutor.RemoteInvokeHandle.Dispose() in /_/src/Microsoft.DotNet.RemoteExecutor/src/RemoteInvokeHandle.cs:line 55
   at Microsoft.Extensions.Hosting.WindowsServiceTester.Dispose(Boolean disposing) in /_/src/libraries/Microsoft.Extensions.Hosting.WindowsServices/tests/WindowsServiceTester.cs:line 147
   at System.ComponentModel.Component.Dispose() in /_/src/libraries/System.ComponentModel.Primitives/src/System/ComponentModel/Component.cs:line 73
   at Microsoft.Extensions.Hosting.WindowsServiceLifetimeTests.ExceptionOnStartIsPropagated() in /_/src/libraries/Microsoft.Extensions.Hosting.WindowsServices/tests/WindowsServiceLifetimeTests.cs:line 85
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr) in /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.cs:line 57

[danmoseley]

@rzikm generally this symptom means a test has hung for reasons specific to that test, right?. So is it useful to have such a general matching pattern? I'm guessing there are many underlying causes here.

The fact that pattern matching can only look at a single line may make it hard to do better..

[rzikm]

@danmoseley I don't think this can be caused by a test failure, if a test fails, then the exception gets propagated to the test method and it behaves as a regular test. So far, we have seen this type of failure happening either on all (or most?) remote executor tests in the test suite or none at all (there is a link at the top of the issue for example test output).

[rzikm]

Maybe we can scope this issue to WebRequestTests, since its those failures which prompted us to create this tracking issue, @liveans what do you think?

[liveans]

Maybe we can scope this issue to WebRequestTests, since its those failures which prompted us to create this tracking issue, @liveans what do you think?

I still believe that this is not specific to WebRequest tests, maybe we should investigate it further, but until then we can decrease the scope of this to WebRequest tests.

[liveans]

This issue was here specifically for HttpWebRequest and WebRequest tests, but there is no hang and issue since 6/25. It was happening constantly on runtime-sanitized before. Closing this as fixed, if we see it again, we can re-open this.

/cc @rzikm