search

derekantrican / MountainProject

A scraper and reddit bot for the website MountainProject.com
57 stars 5 forks source link

[DBBuilder] DBBuilder should retry if it hits a HTTP error #39

Closed derekantrican closed 5 years ago

derekantrican commented 5 years ago

Sometimes the DBBuilder hits an HTTP exception. For instance:

EXCEPTION MESSAGE: One or more errors occurred.

INNER EXCEPTION: System.Net.WebException: An exception occurred during a WebClient request. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at System.Net.ConnectStream.EndRead(IAsyncResult asyncResult)
   at System.Net.WebClient.DownloadBitsReadCallbackState(DownloadBitsState state, IAsyncResult result)
   --- End of inner exception stack trace ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at MountainProjectAPI.Utilities.<GetHtmlDocAsync>d__6.MoveNext() in C:\Users\derek.antrican\source\repos\MountainProject\MountainProjectAPI\Functions\Utilities.cs:line 108
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at MountainProjectAPI.Parsers.<ParseRouteAsync>d__6.MoveNext() in C:\Users\derek.antrican\source\repos\MountainProject\MountainProjectAPI\Functions\Parsers.cs:line 166
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at MountainProjectAPI.Parsers.<ParseAreaAsync>d__3.MoveNext() in C:\Users\derek.antrican\source\repos\MountainProject\MountainProjectAPI\Functions\Parsers.cs:line 91
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at MountainProjectAPI.Parsers.<ParseAreaAsync>d__3.MoveNext() in C:\Users\derek.antrican\source\repos\MountainProject\MountainProjectAPI\Functions\Parsers.cs:line 102
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at MountainProjectAPI.Parsers.<ParseAreaAsync>d__3.MoveNext() in C:\Users\derek.antrican\source\repos\MountainProject\MountainProjectAPI\Functions\Parsers.cs:line 102
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at MountainProjectAPI.Parsers.<ParseAreaAsync>d__3.MoveNext() in C:\Users\derek.antrican\source\repos\MountainProject\MountainProjectAPI\Functions\Parsers.cs:line 102
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at MountainProjectAPI.Parsers.<ParseAreaAsync>d__3.MoveNext() in C:\Users\derek.antrican\source\repos\MountainProject\MountainProjectAPI\Functions\Parsers.cs:line 102
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at MountainProjectAPI.Parsers.<ParseAreaAsync>d__3.MoveNext() in C:\Users\derek.antrican\source\repos\MountainProject\MountainProjectAPI\Functions\Parsers.cs:line 102

STACK TRACE:    at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout)
   at System.Threading.Tasks.Task.WaitAll(Task[] tasks)
   at MountainProjectDBBuilder.Program.BuildDB() in C:\Users\derek.antrican\source\repos\MountainProject\MountainProjectDBBuilder\Program.cs:line 213

The suggestion would be to add a try-catch with retry (maybe with a limit of 5 or 10 retries) to the Utilities.GetHtmlDocAsync method