Unhandled exceptions in Unleash's background process

[EricStG]

Describe the bug When we hit a transient communication error reaching the Unleash server, it generates logs at the error level

To Reproduce Steps to reproduce the behavior:

1. Start a service with the Unleash client library
2. Wait for a transient error?
3. See error

Callstack:

UNLEASH: Unhandled exception from background task 'fetch-feature-toggles-task'.
Unleash.UnleashException: Exception while fetching from API
 ---> System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.
 ---> System.Net.Sockets.SocketException (104): Connection reset by peer
   at System.Net.Sockets.NetworkStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
   at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](CancellationToken cancellationToken, Int32 estimatedSize)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Net.Sockets.SocketAsyncEventArgs.TransferCompletionCallbackCore(Int32 bytesTransferred, Memory`1 socketAddress, SocketFlags receivedFlags, SocketError socketError)
   at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
--- End of stack trace from previous location ---

   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](CancellationToken cancellationToken, Int32 estimatedSize)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](Memory`1 buffer, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.Net.Http.HttpConnection.<EnsureReadAheadTaskHasStarted>g__ReadAheadWithZeroByteReadAsync|40_0()
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Unleash.Communication.UnleashApiClient.FetchToggles(String etag, CancellationToken cancellationToken)
   at Unleash.Scheduling.FetchFeatureTogglesTask.ExecuteAsync(CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Unleash.Scheduling.FetchFeatureTogglesTask.ExecuteAsync(CancellationToken cancellationToken)
   at Unleash.Scheduling.SystemTimerScheduledTaskManager.<>c__DisplayClass3_0.<<ConfigureTask>g__Callback|0>d.MoveNext()

Expected behavior Since those are transient errors and bound to happen, they should not be logged as error, but at warning at best. I would also expect the service to keep retrying, but I can't tell if it is or not at this point

Screenshots N/A

Desktop (please complete the following information):

• OS: Debian (latest ASP.Net Core 8 Docker image)
• Browser N/A
• Version N/A

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context Unleash.Client version 4.1.6"

[daveleek]

Hello @EricStG and thank you for raising this! I had a look at it and it seems to be due to the way we handle synchronous startups. We'll have a look at how we can improve this while leaving flow/functionality unaffected

[daveleek]

Hello @EricStG. We've just released 4.1.8 which should address this issue!