What went wrong with compstate in the smallpeice finals?

[jefftheprogrammer]

We need to find out why the labels were incorrect in the finals and a potential fix.

[PeterJCLaw]

Writing this down while it's hopefully somewhat fresh in my memory. I'm going to include here as much detail as I can recall and would encourage @sedders123 and maybe also @WillB97 to do the same, ideally before reading my notes, so we have a good starting point for a post-mortem (likely a Five Whys). The goal here is to better understand our systems and improve them.

Pre-knockouts:

• James shared with me (Thursday evening) a zip archive of the match files while I was working on https://github.com/sourcebots/compstate-smallpeice2021/pull/6
• (Friday morning) James, Will, myself and a few others were on a Meet call to coordinate remote contributions
• we applied https://github.com/sourcebots/competition-simulator/pull/76 manually by checking out the branch on the simulation runner box
• early in the day we did some testing of the run-knockouts script (https://github.com/sourcebots/compstate-smallpeice2021/pull/7) to get it working on the Windows box in the labs; we also did a full run through (all with competitors' league code) in order to time how long the recordings would take (we fixed an issue with the HDMI cable which sped it up)
• during the league I used the match files I'd had from Thursday evening to push the scores for the league; James validated that these matched his copy of them with the fix applied, so we had good confidence in the league scores
• during the league I was checking manually that the scores in the match files (and thus the compstate) lined up with what was being broadcast, this was due to having manually processed them with https://github.com/sourcebots/compstate-smallpeice2021/pull/6; that all went ok
• At various points we added/updated team names for the teams

Knockouts:

• At various points we added/updated team names for the teams
• At some point in the knockouts team 8 claimed (in the Youtube chat?) that their robot wasn't running their code. We later took this as confirming the theory that it was just the match numbers which were wrong and that the broadcast labelling of the teams in each match was wrong
• Before the broadcast James shared another archive of match files to be put live during the knockouts via Slack; this contained all the match files, though I only copied out the ones needed for the knockouts
• on putting the first of those live (032.yaml) I noticed that the scores didn't match the livestream. The match points were off by one, though had the same winner. On the call we thought this was odd but small enough to ignore, so put the score live.
• When putting the score for the second quarter final live (033.yaml) I noticed that the score (and winner) completely disagreed with the recording. (Aside: oddly I don't recall srcomp validate failing at that point, which I think it ought to have done.)
• Looking at the content of 033.yaml it became clear that its log of actions disagreed with the broadcast match video. Additionally it contained teams which shouldn't have been in that match.
• Further investigation showed that 035.yaml also contained the wrong teams, as though the teams from 33 and 35 were reversed
• We opted to swap those two match files around (which I did -- renaming the files, running srcomp validate, which failed with team mismatches, then also remembering about and fixing the match number record within the file) and those were put live
• However we still couldn't line up the logged actions with the broadcast matches
• I think at some point we spotted that the compstate's projection of who won some of the quarter finals disagreed with the broadcast winners, which we essentially couldn't undo.
• In turn I think that meant that the matches & recordings we had for the semi finals was wrong
• we ended up manually encoding the teams in the semi finals, re-running those, then manually recording the teams in the final by manually checking the scores from those and then recording the final
• After the knockouts James shared another archive which I think contained the full archives directory from the simulation runner box
• I compared one of the knockouts matches from that archive to the broadcast of the same match and found them to be the same, though Will checked some others and thought there was a discrepancy

Unknowns:

• why did the match files disagree with the compbox about which team should be in which match?
  • In theory both came from running SRComp on top of the compstate
  • In theory SRComp won't render the knockouts until all the league match points are present (and from that point the knockout seeding should be immutable unless the league match files, the teams' TLAs or the scorer logic change)
• why did the match files seem to disagree with the broadcast recordings, even before the match number swap?
• did the broadcast matches line up with the videos from the final archive and if not why not?