Remotely hosted running of matches

[PeterJCLaw]

This aims to be a discussion post, as well as (eventually) a milestone target for some work to do in this area.

Following from the automation of the running of matches (#210) it would be great to move the actual running onto a remotely hosted solution, perhaps a Digital Ocean VM or similar. This will further reduce the amount of volunteer effort is needed to run a match session, ideally to the point where running all the matches for a given (league) session is as close to a one-click solution as we can reasonably get it. Just as in #210, I'm deliberately ignoring knockouts here as that would introduce a dependency on Automated Scoring (see also #18), which I'd like to leave out of scope for now.

This would also ensure that we have a single consistent environment for running matches, potentially simplifying running matches outside the formal competition sessions. Given the recent discussions around teams wanting to be able to run "friendly" matches, this feels especially relevant.

Requirements

1. The actual match running must be automated (see #210)
2. Setup of the hosted environment must be simple, either: a. the environment is persistent (in which case a higher setup effort is acceptable), or b. the setup consists of a small number (N < 5) of well documented manual steps
3. The resulting match data (team logs, score logs, recordings) must be easily available
4. Errors during the matches should be reported clearly to the user

Note: I place to update the above list as we think of things -- please do add to it if there are useful things to add.

[PeterJCLaw]

I've had a play this afternoon with running Webots in a Digital Ocean VM, with promising results.

I'm using Ubuntu 20.04 (LTS) x64, which has Python 3.8 by default, for testing. Getting Python 3.7 on Ubuntu is easy, but didn't seem necessary for this test. I'm running in LON1.

My initial approach was using a "Basic" spec machine (specifically 8 GB RAM / 4 vCPUs / 160 GB Disk), however this proved too slow to be realistically useful -- recording a 5 second match took several minutes.

Switching to a CPU-optimised spec (specifically 32 GB RAM / 16 vCPUs) sped things up considerably, so that 5 second recording complete in approximately real-time. It's unclear whether that's a result of the increase in CPU count or due to the CPU-optimised machines getting dedicated cores.

[PeterJCLaw]

Running on a smaller CPU-optimised spec (specifically 4 GB / 2 vCPUs) is fairly slow. Hard to know if the dedicated cores are helping much. In any case, Digital Ocean don't offer "Basic" style specs with high core counts.

In case it's useful I've put the commands I used to do the setup in https://gist.github.com/PeterJCLaw/946de73106a3c20a8ca641f395dee44e.

[WillB97]

Just to clarify, did you have the robots moving during the recording as I've seem a notable performance difference between moving and stationary robots (as much as 100% in some cases)? Also with the timings mentioned here, are they just to reach the "Game Over" message or does this include the time for the movie to finish being created?

[PeterJCLaw]

I was using the example_controller. From the looks of it that does include the robots moving.

[WillB97]

Running on a smaller CPU-optimised spec (specifically 4 GB / 2 vCPUs) is fairly slow. Hard to know if the dedicated cores are helping much. In any case, Digital Ocean don't offer "Basic" style specs with high core counts.

I've done some testing and it appears that core speed has more impact than core count, so I suspect that the dedicated cores are having a large impact.

Here's the stats from my server running at 2.8GHz and the difference that a GPU makes:

Core count	5 second match	20 second match
30	1:47	5:44
24	1:48
16	1:47	5:43
14	1:52
12	1:56
8	2:17
4	3:28
30 w/ GTX650ti	0:10.2	0:25.6
16 w/ GTX650ti	0:09.6	0:24.6
8 w/ GTX650ti	0:09.7	0:23.7
4 w/ GTX650ti	0:09.8	0:24.7

A full match using the GTX650ti GPU takes 2 minutes, 10 seconds.

[PeterJCLaw]

I've done some testing and it appears that core speed has more impact than core count, so I suspect that the dedicated cores are having a large impact.

Nice! Is this on a Cloud platform? If so, which one?

[WillB97]

I've done some testing and it appears that core speed has more impact than core count, so I suspect that the dedicated cores are having a large impact.

Nice! Is this on a Cloud platform? If so, which one?

No, this is just testing on my home server to see the effect of core count.

[PeterJCLaw]

Possibly relevant here: I've just spotted that Webots also publish a Docker image along with docs on using it: https://cyberbotics.com/doc/guide/installation-procedure#installing-the-docker-image

[PeterJCLaw]

I've used Webots' Docker image to augment our CI with a comp match run: https://github.com/srobo/competition-simulator/pull/271. That may be another useful starting point for remote automation of matches.

[WillB97]

I've got the simulator running on the Genesis Cloud GPU system using this installation script: https://gist.github.com/WillB97/e42bf25ac74b63dbdfcfe600cd7dff8a The installation setup is:

wget https://gist.githubusercontent.com/WillB97/e42bf25ac74b63dbdfcfe600cd7dff8a/raw/remote-setup.sh
chmod +x remote-setup.sh
./remote-setup.sh                # the main installation, this will require a sudo password (if configured)
sudo reboot
./remote-setup.sh post_install   # this runs webots  to create initial configurations
rm -r ~/robot_testing            # remove the generated logs from the fake match
./remote-setup.sh settings       # apply high settings and disable camera overlays

Optionally the video rendering can be improved using:

./remote-setup.sh video_mod

On Genesis Cloud this reduced the time to run a match from 3:00 to 1:47.

Matches can then be run with run-comp-match which is aliased to REPO_ROOT/script/run-comp-match. Generally on Genesis Cloud matches take slightly over two minutes which should be fine for our usage.

The box on Genesis Cloud costs 30p/h and you get £50 for registering (or £100 if referred) and we're unlikely to need more than that for this competition since it's only needed for a few hours at each league.

[PeterJCLaw]

Recent versions of Webots support connecting to controller code which is running in Docker containers via a TCP socket, meaning that the options in this space are somewhat greater than they were previously.