Unhandled exception

[sebastian]

I restarted the air and the cloak in a middle of a failed exploration (all running locally – no docker) and got the following exception:

info: System.Net.Http.HttpClient.aircloak-api.ClientHandler[100]
      Sending HTTP request POST http://localhost:8080/api/queries/f237a5ef-b6b0-44d5-ac3f-56f968bb2533/cancel
fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HM136JGBGENK", Request id "0HM136JGBGENK:00000001": An unhandled exception was thrown by the application.
System.Net.Http.HttpRequestException: Connection refused
 ---> System.Net.Sockets.SocketException (61): Connection refused
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Aircloak.JsonApi.JsonApiClient.ApiRequest(HttpMethod requestMethod, Uri apiEndpoint, String requestContent, CancellationToken cancellationToken) in /Users/sebastian/work-projects/explorer/src/aircloak/JsonApi/JsonApiClient.cs:line 320
   at Aircloak.JsonApi.JsonApiClient.ApiPostRequest(Uri apiEndpoint, String requestContent, CancellationToken cancellationToken) in /Users/sebastian/work-projects/explorer/src/aircloak/JsonApi/JsonApiClient.cs:line 286
   at Aircloak.JsonApi.JsonApiClient.CancelQuery(Uri apiUrl, String queryId) in /Users/sebastian/work-projects/explorer/src/aircloak/JsonApi/JsonApiClient.cs:line 251
   at Aircloak.JsonApi.JsonApiClient.PollQueryUntilComplete[TRow](Uri apiUrl, String queryId, DQuery`1 query, TimeSpan pollFrequency, CancellationToken cancellationToken) in /Users/sebastian/work-projects/explorer/src/aircloak/JsonApi/JsonApiClient.cs:line 200
   at Aircloak.JsonApi.JsonApiClient.Query[TRow](Uri apiUrl, String dataSource, DQuery`1 query, TimeSpan pollFrequency, CancellationToken cancellationToken) in /Users/sebastian/work-projects/explorer/src/aircloak/JsonApi/JsonApiClient.cs:line 91
   at Aircloak.JsonApi.AircloakConnection.Exec[TRow](DQuery`1 query) in /Users/sebastian/work-projects/explorer/src/aircloak/JsonApi/AircloakConnection.cs:line 45
   at Explorer.Components.DistinctValuesComponent.Explore() in /Users/sebastian/work-projects/explorer/src/explorer/Components/DistinctValuesComponent.cs:line 81
   at Explorer.Components.ExplorerComponent`1.<get_ResultAsync>b__2_0() in /Users/sebastian/work-projects/explorer/src/explorer/Components/ExplorerComponent.cs:line 11
   at Explorer.Components.DistinctValuesComponent.YieldMetrics()+MoveNext() in /Users/sebastian/work-projects/explorer/src/explorer/Components/DistinctValuesComponent.cs:line 73
   at Explorer.Components.DistinctValuesComponent.YieldMetrics()+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at Explorer.Components.PublisherComponent.PublishMetrics(MetricsPublisher publisher) in /Users/sebastian/work-projects/explorer/src/explorer/Components/PublisherComponent.cs:line 14
   at Explorer.Components.PublisherComponent.PublishMetrics(MetricsPublisher publisher) in /Users/sebastian/work-projects/explorer/src/explorer/Components/PublisherComponent.cs:line 14
   at Explorer.Api.ExplorationRegistry.Remove(Guid id) in /Users/sebastian/work-projects/explorer/src/explorer.api/Exploration/ExplorationRegistry.cs:line 37
   at Explorer.Api.Controllers.ExploreController.Result(Guid explorationId) in /Users/sebastian/work-projects/explorer/src/explorer.api/Controllers/ExploreController.cs:line 89
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

[dandanlen]

Just checking - did this cause any failure of subsequent requests? If so, that would be worrying.

Also, is the issue that exploration was already in a failed state and therefore should no longer be running?

Otherwise, it does pretty much what I would expect if the air server restarts (Connection refused).

There are a few ways we can deal with this if we catch this kind of exception:

1. Implement some sort of retry logic to recover.
2. Try to fail more gracefully (basically, report back that some step of the exploration failed, but that otherwise everything is ok).

Any preference / thoughts?

[sebastian]

Option 2 is my preference. I.e. being more graceful and produce an error that can be reported back to air upon a request when an unexpected crash happens.

[dandanlen]

So it crashed the process? Because by default it should just log the exception and return a 500 error with the exception detail. However the server should still be alive to accept further requests.

[sebastian]

Quite frankly I don't know. So maybe this isn't a problem at all. Feel free to close. I just noticed there was an unhandled exception and thought it might be useful to report.

[dandanlen]

Ok. Clearly there is something to be improved since it definitely looks like something terrible happened to the application (fail!!). I'll keep it open for now until I can check to be sure it doesn't have any wider implications (ie. crashing the system).

[dandanlen]

Closing this - we have made improvements to error handling and will also be adding sentry to (hopefully) collect better diagnostics.