Add "test against gold" feature

[ChandlerSwift]

We have a UI mocked out for this (test_against_gold.html), but it was removed from the menu in 588188538566dd8390b503d11eea7ba2f510afed since it wasn't yet functional.

Open questions (from Slack): @pahp:

• In your mind, would this be simply for running gold to see what it does, or run it like an attack, where we print the output of both gold and the team's executable, and then compare them? I think the first might be slightly easier, but they should be pretty similar, since we've already got all the logic worked out anyway.
• Since users probably want this for fairly quick feedback, do you think it makes sense to prioritize these runs above the "normal" runs (that is, the normal "attack vs. team code" runs). Does that make sense, or can you think of any problems that might cause?
• I figure I'd rate limit these, probably 6-ish per minute, similarly to attacks. I can't think of any issues we'd have without rate limiting other than potentially denial of service, but it seems the safer bet. And 6 per minute is fairly generous, I think.

[pahp]

Hi, sorry for the delay. This just fell out of scope but is easy to answer, I think.

• Just running gold would be awesome. Comparing against the current submitted code would be SUPER AWESOME!
• prioritizing runs makes sense
• yes, rate limiting also makes sense. I agree that 6 is generous. Six per team per minute? Once per minute per user? Really make them think about it could be interesting. Rate limiting for TAG could be a game option, I guess. Nice for newbies, lock it down for oldbies?

Thanks again for all your incredible work.

[ChandlerSwift]

The easiest way to do this seems like to extend the "attack" type to include the tests against gold. If we add a type field to attacks, this could be set to "test" in the test_against_gold case, and "attack" otherwise. Likewise for our Results.

• [ ] Add help text to Test against Gold page
• [x] add type field to attacks and filter on:
  • [x] teams/{id}
  • [x] team ranking (teams/index) -- this one may be automatic
  • [x] attacks
  • [x] disallow attacks/{id} for non-attack types
  • [x] test against gold attack list
  • [x] competition stats page
• [x] ~add type field to results~ (result.attack.type) and filter results on
  • [x] teams/{id}
  • [x] team ranking?
• [ ] Allow to use existing attacks
• [x] Add adding manual tests
• [ ] Allow testing against custom branch (Probably depends on #116)
• [x] Add submission page for above 3 ^ (test.create)
• [x] ~Modify worker to add type=attack flag to existing entries~ -- result.attack.type; more detail to come
• [x] Modify worker to process new entries differently
• [x] Add note to home page about testing against gold
• [x] Add index page for listing previously run attacks (can probably be based on team.show)
• [x] Consider adding a dedicated test.show page?
• [ ] Test that this works when program, say, fails to compile