Fix lots of bugs that occur when playing multiple matches

[msoechting]

Please test this locally!! Leave matches during games early, in warmup, etc. and try to break it. It's pretty solid by now (lots of guards required).