Open jimmyday12 opened 1 month ago
Yeahhh it's not loving the full rescrapes i don't think hey. Looks like it's getting request limited which is fair. For AFLtables maybe we can just do 4 pushes each with 32 years to get a full rescrape - and then we should be fine to go back to just scraping the past year or two. Trying to make sure I'm not creating too much work for you as well!
Will send across 5 pull requests - 1 for each of the 32 year rescrapes - and a final one (weekly-rescrape) that you can merge last just to get back to the normal schedule. Can just merge these one by one to make sure everything works fine?
Haven't digged into the footywire data too much - but there aren't many issues raised with it so don't think we need the full rescrape? Could just change the weekly script to do last season or two? (think that's what it was before splitting into two scripts?) Have changed this to just rescrape this year and last year in the pull requests
Thanks mate, appreciate it. Have approved the first one. I'm assuming we'll need the rescrape to run and finish before then doing the next one so I'll wait for that before approving the 2nd etc
In terms of footywire - I suspect we can just reduce it right down as well
sweet - yep wait 'til the end of each run before merging next. Then I'm happy to check 'em once they've all run to make sure all the issues have been fixed.
lovely - looks like first one ran all good! Quick data check all seems fine as well
Actually gonna be really annoying and ask if you can merge fitzRoy first. Realised there's an issue with jumper.no. being integer not character. Should fix that first in fitzRoy and then rescrape. Have tested this locally and changing to as.character() fixes things. Will pull request the 1897 rescrape again
Probably more for fitzRoy but just making note of it here that the afltables weekly github action is regularly failing if a timeout occurs.
Need to update that function to better handle either retries or silently returning errors and moving to the next URL when this occurs.
I've done this in other contexts using
req_retry
fromhttr2
. The function that we need to update usesrvest
instead so will need to figure out something similar (or replacervest
withhttr2
).https://httr2.r-lib.org/reference/req_retry.html