Proposal: should we scrape Footywire round numbers instead of calculating them?

[cfranklin11]

With all of the complications and exceptions introduced by the irregular fixture this year, I'm wondering if we shouldn't replace the week-based round calculations with scraping round numbers directly from footywire.com. The scraper that I wrote before using fitzRoy did just that, so I have some experience with the tradeoffs involved.

My assessment of the pros and cons of changing to scraping round numbers are as follows:

Pros:

• No longer having to update calculations with the various exceptions introduced by AFL's creative scheduling
  • Even before 2020, irregular matches and rounds have been more common in recent years (all fixes in fix_incorrect_rounds are for the 2010 season or later), so I suspect we'll have to update the fixes once every season or two going forward.
• Simplifies code for data cleaning
  • Manually re-assigning round numbers to a data frame that passed around to various functions isn't easily refactored, because each condition is unique.
  • It's easy to make mistakes in managing the side effects of reassignment (because the functions aren't pure) or accidentally hard-coding an incorrect date or round number.
• Round fixes are difficult to test, and the necessity for future fixes are impossible to test

Cons:

• Complicates web-scraping code
  • Footywire is a little creative with its use of table elements, and the structure of the HTML has clearly been built with a focus on visual aesthetics over ease of data processing.
  • This makes scraping the round numbers and associating them with the matches in the table especially difficult, because the numbers are in their own tr elements, requiring some extra munging to propagate them to the data rows below.

There are probably others that I'm not thinking of, so happy to get feedback on this proposal. I'm also happy to take on the task of implementing it if others think it's worth doing.

[jimmyday12]

@cfranklin11 Yes I think this needs to happen. I have somewhat mitigated issues by adding get_afl_fixture and get_footywire_match_results but ideally we could fix this at the source. I'm very keen to avoid any hardcoded fixes where possible.

I did have a quick look at trying to extract and assign the round to matches but realised it was going to be challenging so haven't revisited it. I'm more than happy for you to take this on!

I've also got some general ideas for re-structuring the functions in the package moving forward, I might share that in a new ticket for feedback and discussion. It shouldn't affect anything here but just figured I'd mention it