search

dotnet / crank

Benchmarking infrastructure for applications
MIT License
975 stars 103 forks source link

Job gets stuck in "initializing" #697

Open sebastienros opened 7 months ago

sebastienros commented 7 months ago 
[08:52:59 INF] Processing job 'application' (36) in state Initializing
[08:52:59 INF] Creating attachment: clrjit.dll
[08:52:59 INF] Uploading archive: oqjfzh21.30x.zip
[08:52:59 INF] Received uncompressed file content
[08:53:00 INF] Creating attachment: coreclr.dll
[08:53:00 INF] Processing job 'application' (36) in state Initializing
[08:53:00 INF] Uploading archive: 5fbaajpa.ko2.zip
[08:53:00 INF] Received uncompressed file content
[08:53:01 INF] Processing job 'application' (36) in state Initializing
[08:53:02 INF] Processing job 'application' (36) in state Initializing
[08:53:03 INF] Processing job 'application' (36) in state Initializing
[08:53:04 INF] Processing job 'application' (36) in state Initializing
[08:53:05 INF] Processing job 'application' (36) in state Initializing
[08:53:06 INF] Processing job 'application' (36) in state Initializing
[08:53:07 INF] Processing job 'application' (36) in state Initializing
[08:53:08 INF] Processing job 'application' (36) in state Initializing
[08:53:09 INF] Processing job 'application' (36) in state Initializing
[08:53:10 INF] Processing job 'application' (36) in state Initializing
[08:53:11 INF] Processing job 'application' (36) in state Initializing
[08:53:12 INF] Processing job 'application' (36) in state Initializing
[08:53:13 INF] Processing job 'application' (36) in state Initializing
[08:53:14 INF] Processing job 'application' (36) in state Initializing
sebastienros commented 7 months ago

Controller:

[08:53:00.480] Uploading C:\Users\andya\AppData\Local\Temp\5fbaajpa.ko2.zip (5,812KB)
[08:53:06.158] Could not ping the agent on 'asp-perf-win:5001', retrying ...
[08:53:06.737] Could not ping the agent on 'asp-perf-load:5001', retrying ...
[08:53:09.799] Could not ping the agent on 'asp-perf-load:5001', retrying ...
Unhandled exception. System.InvalidOperationException: An error occurred while uploading a file.
---> System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine..
---> System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Microsoft.Crank.Controller.JobConnection.UploadFileAsync(String filename, String uri, Boolean gzipped) in /_/src/Microsoft.Crank.Controller/JobConnection.cs:line 1056
   --- End of inner exception stack trace ---
   at Microsoft.Crank.Controller.JobConnection.UploadFileAsync(String filename, String uri, Boolean gzipped) in /_/src/Microsoft.Crank.Controller/JobConnection.cs:line 1062
   at Microsoft.Crank.Controller.JobConnection.StartAsync(String jobName) in /_/src/Microsoft.Crank.Controller/JobConnection.cs:line 305
   at Microsoft.Crank.Controller.Program.<>c__DisplayClass54_1.<<Run>b__6>d.MoveNext() in /_/src/Microsoft.Crank.Controller/Program.cs:line 887
--- End of stack trace from previous location ---
   at Microsoft.Crank.Controller.Program.Run(Configuration configuration, String[] dependencies, String session, Int32 iterations, ExcludeOptions exclude, IEnumerable`1 scripts) in /_/src/Microsoft.Crank.Controller/Program.cs:line 812
   at Microsoft.Crank.Controller.Program.<>c__DisplayClass53_0.<<Main>b__1>d.MoveNext() in /_/src/Microsoft.Crank.Controller/Program.cs:line 672
--- End of stack trace from previous location ---
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.ExecuteAsync(String[] args, CancellationToken cancellationToken)
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
   at Microsoft.Crank.Controller.Program.Main(String[] args) in /_/src/Microsoft.Crank.Controller/Program.cs:line 718
[08:53:13.393] Could not ping the agent on 'asp-perf-win:5001', retrying ...
sebastienros commented 7 months ago

Upload files might need some retry logic in case the VPN is lost temporarily.