Automate running of matches

[PeterJCLaw]

This aims to be a discussion post, as well as (eventually) a milestone target for some work to do in this area.

It would be great to automate the running of matches. This will reduce the amount of volunteer effort is needed to run a match session, ideally to the point where running all the matches for a given (league) session is as close to a one-click solution as we can reasonably get it. I'm deliberately ignoring knockouts here as that would introduce a dependency on Automated Scoring (see also #18), which I'd like to leave out of scope for now.

I'm therefore aiming to capture the things we want from competition matches so that we know what to automate. I hope that we can then also have a discussion about how best to achieve those outcomes, potentially in different ways than we have in the past.

Requirements

1. Teams code needs to be put somewhere so that the right robot controller can run it
2. The simulated world should be fresh at the start of the match
3. The simulated match should be recorded (animation and video) a. ideally this would be tagged with the match id, for our convenience
4. Errors during the match should be reported clearly to the user
5. Teams log output needs to end up somewhere so that we can share it with them afterwards a. it should be clear to the team which log came from which match, and b. ideally this would group by team for our convenience

Note: I place to update the above list as we think of things -- please do add to it if there are useful things to add.

[PeterJCLaw]

Existing solutions

(as of 2020-11-07)

1. Currently we have a prepare-comp-match script which handles putting the team code in the right places.
2. We assume that the simulation is started afresh each time from a clean instance of the repo
3. When run in comp mode we automatically capture video and animation a. no solution for match ids yet :(
4. We rely on the user spotting errors in Webot's console :(
5. Robot logs end up next to the respective code a. no solution for match ids yet :( b. being next to the code sort-of allows this, but it's not the easiest to map back to TLAs :(

185 aims to fix 5. b., however ends up quite coupled to SRComp internals and doesn't solve 5. a.

Otherwise we currently handle things manually.

[PeterJCLaw]

Suggestion for future direction

I think we would benefit from moving towards explicit configuration of the simulator in comp mode. At the moment we have a number of aspects which work separately to achieve parts of this, but which I think we should centralise. I would still like it to be possible to control each of the individually (certainly I don't want comp mode to become a single switch which is the only way to control lots of separate behaviours), even if comp mode does change a lot of defaults.

In #204 I introduced an environment variable to control the location of the directory containing the zone directories. I had expected this to be useful for automating the running of matches too -- for example by allowing us to pre-create match directories containing zone directories and then simply point the simulator at each when the time came to run that match.

A possible extension to this idea would be to use an environ based configuration file (think starlette.config or django-environ) which contained defaults easily overridden by environment variables. I'm not sure whether the "mode" of the simulation would also be controlled by this, however it's possible that it could be. We'd need to ensure it didn't become too hard for competitors to enable comp mode though.

[PeterJCLaw]

Concrete proposal

For the immediate term I think we should stick to assuming that we'll run one match at a time and aim to get that as close to a one-click operation as we can, without doing anything which moves us away from overall automation. In the end we do want to be able to run a single match as a one-click anyway.

Two things then will enable this:

• Introduce a wrapper around webots' invocation to set environment variables based on configuration. This can live in this repo, but should aim to not have any dependencies on the Python controller code.
• Introduce environment variables for conveying information from that script into the simulation.

The environment variables would be:

• LOG_PATH with handling for that optionally containing placeholders of {zone_id} and {when}
• RECORDING_PATH with handling for that optionally containing a placeholder of {when}

Both would default to their current (eventual) values, expressed through suitable default values for the environment variables.

This would allow us to easily bake in arbitrary naming content to the log and recording files as well as to put them where we like. Judicious use of symlinks could allow us to further control the eventual location of the files such that zone numbers are converted to TLAs for example.

The role of the script would be:

• be passed the TLAs per zone, likely much as prepare-comp-match is now, plus the match id and the (general) output directory
• configure the arena directory (TLA/zone code, comp mode marker, etc.)
• configure the environment (so that the recording & logs end up where we want them)
• run webots
• validate from the user that the match completed successfully (not sure about this -- merely letting the user close Webots manually rather than killing the script may be enough)
• tidy things up afterwards as needed

It is also reasonably straight-forward to see how this sort of script could be further wrapped by one which consumed a whole session of match data from SRComp and ran all the matches in one go. I suggest we leave that out of scope for now though.

[PeterJCLaw]

A possible extension to this idea would be to use an environ based configuration file (think starlette.config or django-environ) which contained defaults easily overridden by environment variables.

https://pypi.org/project/python-dotenv/ looks like it provides effectively this, without being coupled to anything else.

[PeterJCLaw]

I've started exploring this in match-automation. That branch is likely to be rebased all over the place, though it may give an idea of the implementation direction.

[WillB97]

A possiblity for automating multiple matches could be to use the compstate to provide the match number and TLAs. Additionally scoring could be run at this point and the compstate updated.

[PeterJCLaw]

https://pypi.org/project/python-dotenv/ looks like it provides effectively this, without being coupled to anything else.

Of course using that would mean that we'd need to sort the issue of distributing third party dependencies.. which we haven't yet.

[PeterJCLaw]

One complication appears here -- while Windows does support symlinks, it seems that ordinary users don't by default have permission to create them. Specifically, running os.symlink('foo', 'bar') is met with an OSError number 1314 on Windows. Developer mode in Windows 10 can apparently sidestep this, though only for Python 3.8+ (we're using 3.7). This means that the approach I'd hoped to use (using symlinks to map back from zones to TLAs) likely won't work on Windows.

Since we do want to be able to run competition matches on Windows, we may need a new approach for this.

[PeterJCLaw]

A possiblity for automating multiple matches could be to use the compstate to provide the match number and TLAs. Additionally scoring could be run at this point and the compstate updated.

Yes, I definitely see this being driven by data from the compstate. I'm deliberately trying to avoid coupling the simulator to having a compstate (or SRComp) around though as that feels likely to complicate development or other one-off running in comp mode. I definitely see a top-level wrapper which pulls in compstate data and runs the simulator as the end-goal here, however I'm expecting that to be as thin as possible.

[PeterJCLaw]

Current direction update

With #256 (and the preceding PRs it depends on) I'm heading in a slightly different direction than I'd originally outlined here. This has mostly been driven by a desire for this to work on Windows, where we don't have symlinks.

I've also ended up essentially introducing a single configuration file (#190) in the form of a Proton-compatible match.json in the root of the arena directory. This turns out to be quite useful in development too as the settings for a given Arena are now persistent :)

The result is that the unit-of-configuration ends up being the Arena. Given that Arenas are trivial to specify through the ARENA_ROOT environment variable having everything else as files within that directory is feeling like a good solution that works well both in development and for running actual matches (the latter being heavily automated).

The remaining challenge is where the output files should end up. I'm currently heading in the direction of having a single persistent directory (the one which contains the teams' TLA.zip archives) which is both the source and sink of information. At the end of a series of matches I'm expecting that directory to look like this:

$ tree -a league-1-2021-01-16
league-1-2021-01-16
├── ABC
│   ├── log-match-42-zone-0.txt
│   └── log-match-43-zone-1.txt
├── ABC.zip
├── DEF
│   ├── log-match-42-zone-1.txt
│   └── log-match-43-zone-0.txt
├── DEF.zip
├── matches  # These are our 'match.json' files, suitably renamed
│   ├── 042.yaml
│   └── 043.yaml
└── recordings
    ├── match-42.html
    ├── match-42.json   # Note: this is webots record of the simulation data, not our match.json file
    ├── match-42.mp4
    ├── match-42.x3d
    ├── match-43.html
    ├── match-43.json
    ├── match-43.mp4
    ├── match-43.x3d
    └── textures
        └── ...

This would allow the directory as a whole to be archived for storage or sharing. Copying the match data to the compstate is also trivial -- you copy the matches directory to within the proper arena directory within league, knockouts or tiebreaker.

[WillB97]

Is the intention to pass a full path to the robots' for logs or to move the logs after the match completes? If this can be reasonably done, it seems logical for things to be written directly to the final location rather than to require an additional step.

Having the match.json as the de facto configuration file is a good way to pass information to all controllers during the simulation. This folder structure looks great for organising all the files, since teams' logs are now grouped we just need to figure out how to return them to the teams.

[PeterJCLaw]

Is the intention to pass a full path to the robots' for logs or to move the logs after the match completes? If this can be reasonably done, it seems logical for things to be written directly to the final location rather than to require an additional step.

My original plan here was to specify the full path and make use of symlinks to redirect from the zone number to the TLA directory. However as symlinks aren't really a think on Windows that's not an option. While we could definitely configure each of the zones' log paths independently that doesn't feel like it's a particularly neat solution.

I don't think it's a big deal to have an extra step which copies the files afterwards, especially as it keeps the behaviour in development mode close to that in comp mode.

This folder structure looks great for organising all the files, since teams' logs are now grouped we just need to figure out how to return them to the teams.

Yup, I'm hoping that that's an easy step. I'd imagined a while back that that could be done using the code-submitter, though obviously what we do here is relatively independent of that.