Galaxypedia-Wiki / ketchupbot-updater

epic galxy pedia updator
MIT License
1 stars 0 forks source link

Completely overhaul ketchupbot #130

Closed smallketchup82 closed 1 month ago

smallketchup82 commented 1 month ago

Closes #111 Closes #95 Closes #124

HttpClient

Currently, classes create their own HttpClients, which poses numerous different problems:

  1. We introduce a lot of code duplication (for example, with the user agent)
  2. We cannot substitute the HttpClient in the classes for use in Mocking
  3. Lifetimes of HttpClients are not handled properly and can grow in size (remember: it's creating a new HttpClient every time a new class is created)
  4. Setting up exponential backoff is exponentially more difficult

This PR moves HttpClient creation to Dependency Injection, alleviating all of those issues. HttpClients for those classes are passed through the constructor, allowing us to use Dependency Injection to pass HttpClients to classes that need them, and it also allows us to pass mock HttpClients for use in testing.

HttpClientFactory will also manage the lifetime of HttpClients properly. HttpClients are now transient and a new instance is created every time the respective class is created, with a caveat: Their resources have different lifetimes. It's hard to explain, but in a nutshell, the HttpClients will share, cache, and pool resources so as to not cause issues when too many are created.

Exponential Backoff

This PR implements polly in a much better fashion than in the old PR.

Galaxy Info

In my experience, the Galaxy Info API tends to go down often and for hours at times. So when approaching retrying here, we need to consider a couple of factors:

I decided to answer these questions myself. I believe that we can wait up to 5 minutes after a request is made before deeming it a lost cause. I also believe that we should retry exponentially as much as we can within that 5 minutes, adding in a bit of jitter as a cherry on top.

So, that's how I plan on approaching retrying requests to Galaxy Info. When we make a request and get an unsuccessful response, we will continue to retry it. If one of those tries end up working, excellent. If not, we have 5 minutes to continue retrying. If that 5 minutes gets used up with no success, we give up and return an error.

Galaxypedia

On the other hand, the Galaxypedia's uptime is very consistent. Here, we decide to ditch the above approach and use a different retry method.

In short, we retry 3 times exponentially with a bit of jitter between each attempt. If all of those fail, the request is a failure. We tend to make many requests to the Galaxypedia in short succession, so we do not want to wait long to reach an outcome.

Formatting

Did a full jetbrains resharper formatting sweep.

Smaller changes

Can't think of any except for making the cache for ApiManager optional