search

perrycate / tournamap

Website to visualize Smash Ultimate tournaments near you
https://tournamap.gg
GNU General Public License v3.0
9 stars 6 forks source link

Handle paginated data from start.gg properly #58

Closed perrycate closed 1 year ago

perrycate commented 1 year ago

Currently our fetch script doesn't bother interacting with start.gg's pagination - we can request a page by index, and each page has 500 tournaments, so we just assume that there are fewer than 5,000 tournaments (seems to be the case in general), request 10 pages, and call it a day.

This has a few downsides:

  1. We will miss data if there are too many tournaments.
  2. Our newly-added error handling will report the same error multiple times if something affects each request (for example, a stale auth token).
  3. We make more requests than necessary if there are fewer than 5k tournaments (though for our purposes this doesn't actually matter).

We should instead request one page at a time and check the total number of pages included in the PageInfo (docs here, though I believe we're already requesting it) before making each request. This means we will make exactly as many requests as are necessary, and can return early anytime if we encounter an error.

dknham commented 1 year ago

Hello, I can help with this if you don't mind!

A simple solution would be to send a request to get the total number of pages, and use this number as the upper limit of the range (instead of hardcoded value of 10). Let me know your thoughts!

perrycate commented 1 year ago

Sure thing, help is always welcome!

That solution seems reasonable! Let me know if you have any questions! Also, there are some moving parts adjacent to this in PR #57 - you might want to wait until that's in (hopefully very soon!) before working on this to avoid merge conflicts.

perrycate commented 1 year ago

That PR is now merged!

dknham commented 1 year ago

PR link: #59

perrycate commented 1 year ago

Fixed by PR #59 . Thanks again!