Suggestions for the AIIDE competition and/or ladder

[chriscoxe]

I have finally got around to listening to the Undermind podcasts and noticed that in one episode, you asked long-time bot developers for suggestions for StarcraftAITournamentManager for AIIDE. Pasting the relevant discussion from Discord:

I have some suggestions for what's missing from current tournament setups:

1. Rules/logic for Current Working Dir and run_proxy.bat

Refer to the e-mail conversation I had with Dave and Rick about this at the end of Oct '18. Clarify in the Rules page whether the new TM software will set the current working dir to the Starcraft dir (e.g. C:\TM\Starcraft) or the AI dir (e.g. C:\TM\Starcraft\bwapi-data\AI), for DLL bots and non-DLL bots like EXEs/JARs. Afaik, sc-docker always sets the current working dir to the Starcraft dir (not the AI dir) then starts bwapi-data/AI/.dll or bwapi-data/AI/run_proxy.bat. Should non-DLL bots submit a run_proxy.bat file, and if so, should it change the working dir at all (if so, to what absolute path or what relative path?)? Or for non-DLL bots, perhaps authors should submit a run_proxy.bat file that either doesn't set/change it, or sets it to a particular environment var if it exists (e.g. AIIDE_STARCRAFT_DIR), otherwise the bot is allowed to set it to whatever they like (so that it will work on their personal dev/test environments and AIIDE without editing). Even UAlbertaBot is confusing - its' run_proxy.bat does "cd c:\TM\Starcraft\bwapi-data\AI" then "c:\TM\Starcraft\bwapi-data\AI\UAlbertaBot.exe" then its' config file sets the "ReadDirectory" setting to "../read/" and "WriteDirectory" to "../write/". Even though the Rules page says to use "bwapi-data/read" and "bwapi-data/write". If this isn't fixed or clarified (ideally fix StarcraftAITournamentManager to follow the Rules page, which is also how sc-docker works), we will keep seeing broken bots and teething problems in the various competitions. It is a pity this single issue has caused so many broken/weak bots in past AIIDE & CIG competitions.

PurpleWave / Cheese Maven: I think StarCraft directory is intended to be the one true working directory just that implementations of run_proxy.bat in tournaments (namely, CIG) have not achieved that consistently

Afaik, the latest dev version of StarcraftAITournamentManager sets it to the .../Starcraft/bwapi-data/AI folder (not the .../Starcraft folder), so unless that is fixed, the run_proxy.bat file must be set up to include a pushd or cd command that changes the working dir (to e.g. C:\TM\Starcraft or ../..) in order to get the bot to work properly (e.g. find BWTA2 map data at the correct location, and so that File I/O uses the correct paths to the read and write folders etc).

The main change would be to edit ClientCommands.java and change the line: WindowsCommandTools.RunWindowsCommand("cd " + ClientSettings.Instance().ClientStarcraftDir + "bwapi-data\\AI & "+ ClientSettings.Instance().ClientStarcraftDir + "bwapi-data\\AI\\run_proxy.bat", false, false); to: WindowsCommandTools.RunWindowsCommand("cd " + ClientSettings.Instance().ClientStarcraftDir & "+ ClientSettings.Instance().ClientStarcraftDir + "bwapi-data\\AI\\run_proxy.bat", false, false); ... and also change UAlbertaBot accordingly if you still want UAlbertaBot to work. I don't know whether any other old AIIDE bots would need updating (possibly just some of their run_proxy.bat files). On the AIIDE competition website (maybe on the Rules page) I also suggest showing a sample run_proxy.bat file for a JAR bot and a sample run_proxy.bat for an EXE bot.

2. Ensure the Tournament Module allows some permissions for some versions of BWAPI

Update: this point has been moved into #32.

After the bug in point 7 is fixed, rebuild the TournamentModule.dll's (for BWAPI 4.2.0 and 4.1.2 at least - I wouldn't bother fixing 4.0.1 Beta, and I wouldn't bother changing 3.7.4 which doesn't allow any permissions) to allow bots to do setLatCom(false) and setCommandOptimizationLevel(a value other than 1, i.e. especially 0 or 2, but perhaps some other authors might want 3 or 4). They don't affect the opponent bot at all - it only affects what info BWAPI returns to you, and some bots rely on them. The default should probably stay as 1, but I think bots should be allowed to set it to 0 and other values if they want to, even though a value of 0 may increase the bot's APM and replay file sizes, because some bots rely on them and currently there is no way for a bot to check what the command optimization level is. And anyway, currently, BWAPI version 4.2.0/4.1.2/4.0.1 Beta bots have always done this until now, due to the bug, so I don't think it's a problem. I suggest to also allow leaveGame(), so that bots can leave the game to indicate they are resigning/surrendering in lost positions, or in bugged states to attempt to avoid crashing later (and this may help address https://github.com/bwapi/bwapi/issues/719 ). This might also help increase the overall game rate, so long as Starcraft or bots don't/rarely crash when leaving, because bot-vs-bot endgames can often take a lot of time if the winning bot is weak/slow. Personally, I don't care whether sendText/printf/setTextSize are allowed, so I would probably disallow them on anti-collusion grounds and in case they help to increase performance, but I think the others should be disallowed.

Note: until the bug noticed in point 7 is fixed, the Tournament Module for BWAPI versions 4.2.0, 4.1.2 and 4.0.1 Beta actually allows all permissions (even for DLL bots, not just for non-DLL bots). This point should be addressed after point 7 is fixed.

3. Game Speed slider setting in game lobbies

Update: this issue has been moved to #37.

At least a few months before the competition (so we have time for sufficient training/testing), tell us in the Rules page whether the Game Speed slider in the game lobbies will be set to Normal or Fastest (i.e. LF3 or LF6 respectively). I added a feature request to BWAPI to make organizers' management easier for this (https://github.com/bwapi/bwapi/issues/807). If you want to use "Fastest", I suggest you ask as early as possible whether there are any objections from the big teams like Facebook and Sumsung and other serious competitors, because bots that rely on ML training might need a lot of time/resources to be re-trained.

4. Ability to disable bots (not just ignore their results)

Ability to disqualify a bot (manually at least, perhaps also automatically based on some thresholds) so that it doesn't play any of the remaining games, e.g. if it crashes/times out too much. I know there's already a feature to exclude a bot from the results summaries, but afaik it still plays all the remaining games. I could be wrong. I don't want this ability for my testing - it's just an idea you might find helpful.

5. Better checks for why some games fail to start

Better checks for whether the TM actually handed control to each individual bot for frame 0 and whether each individual bot actually returned control to the TM on frame 0, so that it is clear whether games that didn't start properly are a problem before each bot was started (i.e. not either of the bots' fault), or a problem after each bot tried to start. I haven't investigated whether the following is how StarcraftAITournamentManager already works, but please check the following cases:

• If HeroBot started ok and returned control to the TM on frame 0, but the TM never got as far as starting the call(s) to MegaBot's binaries for frame 0, HeroBot should count it as a win, and MegaBot's "total games played" shouldn't count that game. Or if you want symmetry, count is as a loss for MegaBot.
• If HeroBot started ok and returned control to the TM on frame 0, and the TM started the call(s) to MegaBot's binaries for frame 0 but the TM never got as far as passing control to MegaBot for frame 0 (e.g. missing mandatory DLL dependencies, so MegaBot didn't get as far as entering its onStart() function), HeroBot should count it as a win, and it's up to you whether it should be counted as a loss for MegaBot (if you want symmetry) or MegaBot's "total games played" shouldn't count that game. In this case, it's unclear whether to blame MegaBot or an environmental problem caused by the TM or Starcraft/bots from old games (e.g. out of memory from a old processes that are still running for whatever reason).
• If HeroBot started ok and returned control to the TM on frame 0, and the TM passed control to MegaBot for frame 0 but MegaBot didn't return control back to the TM for frame 0 (this happened a lot for some bots like MegaBot), HeroBot should be assigned a win (rather than ignoring the game entirely), and MegaBot should be assigned a loss.

I don't know whether there's any way in the TM to distinguish between the last two cases though. In all cases, try to distinguish the various cases as much as feasible in a field that stores the reason for the result for that bot.

6. Request a new official BWAPI 4.x release

Update: this point has been moved into #34.

Request a new official BWAPI 4.x release (4.2.0 was released most recently, so it would be something like 4.3.0) and build the corresponding TournamentModule.dll (& add it & its source code to GitHub). A few bots are probably now using a more recent dev version of BWAPI than 4.2.0 (post-4.2.0 includes improvements to BWAPI's latency compensation logic, among other things).

7. TournamentModule.dll allows all permissions for BWAPI version 4 (e.g. 4.2.0, 4.1.2, 4.0.1 Beta)!

Update: this point has been moved into #32.

Due to a change to a BWAPI function's argument type as part of the first release of version 4 that was not correspondingly migrated into the Tournament Module (i.e. the TournamentModule::onAction function's first argument type was changed from an into to a BWAPI::Tournament::ActionID), permissions like enableFlag(CompleteMapInformation), setLocalSpeed(42) are not enforced by TournamentModule.dll, because the default implementation of the function that is defined in BWAPI is being used, which always returns true. This is why the Sling bot was able to successfully do setLocalSpeed(42) and setFrameSkip(0) until Sling's source code was patched, and the Tyr/TyrProtoss bot was able to successfully do setLocalSpeed(10) in the AIIDE and CIG competitions, which slowed down the runs. If any bot that uses BWAPI 4 had cheated by doing things like enableFlag(CompleteMapInformation), the tournament module wouldn't have stopped them. Note that this is true for non-DLL bots and for DLL bots (originally I thought it only affected non-DLL bots, but after investigating and finding the cause of the bug I realized it affects DLL bots too).

8. Add a Win Percentage Per Map Cycle Graph

In the summary results, add a graph for "Win Percentage Per Map Cycle Graph". So if there are 10 maps, rather than ~100 points the X axis, there would only be 100/10 = 10. For each point on the X axis, it would only consider the games for that particular cycle through the set of maps (not consider the earlier cycles too). This avoids the noise caused by bots being better on some maps than others, and makes it easier to identify turning points in the progress of a bot's learning.

9. Add a Win Percentage Over Time by Map Cycle Graph

In the summary results, add a graph for "Win Percentage Over Time by Map Cycle Graph". For each point on the X axis, it would only consider the games for that map cycle and the earlier cycles combined. This avoids the noise caused by bots being better on some maps than others, and shows the rankings as-at the end of each map cycle.

10. Add an ability to pick race randomly out of Terran/Protoss/Zerg in the game lobby

I am not suggesting to forbid picking "Random" race in the game lobby - just to allow bots to specify that the TM should pick randomly out of Terran/Protoss/Zerg from the drop-down in the game lobby. This makes the management easier for authors so they can easily write and update a single bot that supports playing as different races (rather than needing to set up & update one bot that always plays as Terran like tscmoo (Terran), one bot that always plays as Protoss like tscmoop, one bot that always plays as Zerg like tscmooz). Note: in terms of game results records and File I/O, it is important to distinguish between picking "Random" race from the drop-down and picking a specific race out of Terran/Protoss/Zerg, because the meta is quite different when the opponent knows your race at the very start of the game - they don't need to scout your race or common proxy locations to help rule out some of the various possible early game attacks like 4-6 pool, offensive sunken rush, cannon rush, BBS or bunker rush etc. Some human players despise (& I get the feeling some bot authors dislike) that the meta is different against Random race players that exploit the fact that the opponent does not know their race, compared with knowing the opponent's race. I know that Dave doesn't see this as a concern for AIIDE until a bot actually relies on it, but personally I think this feature would be handy for future-proofing in case it becomes a problem sooner than expected. No big deal.

11. Add support for BWAPI 3.7.5

Add support for BWAPI 3.7.5. I wouldn't use it, but some of the GOAL-based bots in particular, might.

12. Add remaining SSCAIT & CIG maps with BWTA2/BWTA data

Add the map files and BWTA2 & BWTA data for the SSCAIT and CIG maps. E.g. create "maps/sscai" and "maps/cig" folders for the map files within each ZIP file in the "server/required" folder that corresponds to a particular BWAPI version, and similarly for the "bwapi-data/BWTA" and "bwapi-data/BWTA2" folders. This will make it easier for the CIG organizers to run their competitions, and would be very convenient for bot authors that want to train their bot. I know there is occasionally talk of changing the map pool for some comps/ladders, but I wouldn't hold my breath. The map files and BWTA2 & BWTA data for the remaining CIG maps can be downloaded from here. The map files for SSCAIT can be downloaded from here but I am not aware of an official source for their BWTA2/BWTA data. I could send you the files I have, or I could fork & add what I have. I probably downloaded them from another bot, or I made a bot generate them.

13. Separate storage

It's quite time-consuming to manage the "RequiredBWAPI.zip" files, e.g. to add a support for a new BWAPI version or add maps to all supported BWAPI versions. Most of the contents don't seem to vary according to the BWAPI version, so a feature that might make it easier for organizers and bot authors to manage in future is to separate the common files to a separate ZIP file. I.E. almost everything except the BWAPI-related DLLs, bwapi.ini, "Multiple Instance Hack.bat", TournamentModule.dll, Broodwar.map, bspatch.exe, starcraft.bsdiff, maybe injectory.x86.exe (not sure whether this depends on BWAPI version), and SNP_DirectIP.snp. I suggest that almost all of the files that are used on client VMs except the actual bot files/data are installed on client VMs. Some automation could be added that recreates the bwapi-data folder etc and copies the files into place each game, rather than downloaded every time a client downloads a bot. This avoids wasting a lot of time downloading a lot of data from the server every game. Some folders should be re-created every game though, e.g. the bwapi-data folder, and maybe the characters folder, and maybe older BWAPI versions like 3.7.4 or 3.7.5 have some special files/folders (not sure). Likewise, the files associated with a particular version of BWAPI could all be installed somewhere on the client VMs and copied into place every game, depending on the BWAPI version used by the bot.

14. Add dependencies for BWAPI/bot-related libraries

Install onto the client VMs' Starcraft folder the DLLs etc for BWAPI-related libraries that bots commonly depend on such as the various versions of BWAPI bridge (bwapi_bridge1_2.dll, bwapi_bridge2_3.dll, bwapi_bridge2_4.dll, bwapi_bridge2_5.dll, bwapi_bridge2_6.dll), QtCore4.dll, QtGui4.dll, and the files sc-docker has at https://github.com/Games-and-Simulations/sc-docker/tree/master/docker/dlls. And config files used by some bots like Aiur to configure where the correct locations of the read and write folders are (directories.txt, file_settings.txt) to stop them updating files in the Starcraft folder itself. This would really help to more easily set up and manage client VMs and avoid a lot of headaches trying to get bots to start / work properly due to missing DLL dependencies or File I/O configuration issues. I can add the files I have to a fork if you like.

[davechurchill]

Thanks Chris! Appreciate the feedback. We'll get to this list as the work on the ladder progresses

[chriscoxe]

FYI, I edited the Description to add suggestion # 10.

[chriscoxe]

Added more suggestions to the Description.

[chriscoxe]

@davechurchill @richard-kelly it might save you quite a bit of time/effort if I go through what I have and make some updates to a fork based on what I have. If you'd like me to do that, please let me know. If you're agreeable, I am also wondering whether I could add map/BWTA2/BWTA files for some other maps that the community has been using, e.g. AITT, pro season maps used for AIST, and perhaps the BGH map that SSCAIT uses on April Fool's day. The downside is that it does take longer for clients to download the ZIP file from the server because each BWTA or BWTA2 file is about 1-2 MB uncompressed. I added just the SSCAIT and CIG data to my "Required_BWAPI_420.zip" file and it is now 7,351 KB rather than 2,012 KB. It would be even larger with the other maps/data. If you want to keep the competitions lean and mean, I could create a separate ZIP file and people could rename the ZIPs if they want to use it, e.g. "Required_BWAPI_420_extras.zip". CIG could rename it and delete the map/data files for the non-CIG maps so their comp is lean and mean. Or implement the separate storage feature in suggestion # 13 so that the maps etc don't need to be downloaded every game.

[chriscoxe]

After some investigation, debugging, fixes and testing in my fork (I will push to my fork soon), I have updated the Description for points 2 and 7 accordingly and moved them into a separate issue: #32. Similarly, I moved point 6 into #34.

[chriscoxe]

I have moved point 3 into a separate issue: #37.

[richard-kelly]

Summary:

• Resolved: 1, 2, 3, 4, 6, 7
• Will do soon: 11, 13
• Will look at/do later: 5, 8, 9, 10, 12
• Unsure: 14

1. Rules/logic for Current Working Dir and run_proxy.bat

I agree with you on this and have made this change so that the working dir is the Starcraft dir.

2. Ensure the Tournament Module allows some permissions for some versions of BWAPI

Fixed by @chriscoxe

3. Game Speed slider setting in game lobbies

This will be Normal, to be updated soon on the competition website

4. Ability to disable bots (not just ignore their results)

I made some changes to this feature so that now excluding a bot prevents future games with the bot from being played, and removes it completely from the results summary and detailed results. Server restart is required, obviously, and you can re-parse the results of a completed tournament with the standalone result parser.

5. Better checks for why some games fail to start

I accounted for everything I could think of without changing the TournamentModule when I added the Game "End Type" that now appears in the detailed results. I'll have another look at this.

6. Request a new official BWAPI 4.x release

BWAPI 4.4.0 has been released and is now supported.

7. TournamentModule.dll allows all permissions for BWAPI version 4 (e.g. 4.2.0, 4.1.2, 4.0.1 Beta)!

Fixed by @chriscoxe

8. Add a Win Percentage Per Map Cycle Graph & 9. Add a Win Percentage Over Time by Map Cycle Graph

I'll add these.

10. Add an ability to pick race randomly out of Terran/Protoss/Zerg in the game lobby

Shouldn't be hard to do, but I'm missing how this is easier for authors than just picking "Random" in the lobby.

11. Add support for BWAPI 3.7.5

I'll do this soon.

12. Add remaining SSCAIT & CIG maps with BWTA2/BWTA data

I will look at this after doing 13.

13. Separate storage

I'll do this soon.

14. Add dependencies for BWAPI/bot-related libraries

I'm not sure if you're asking for something to be added to the project, since we don't provide the SC environment. Do you want us to have a list of libraries that will be available in the competition?

[chriscoxe]

Thanks! Comments:

10. Add an ability to pick race randomly out of Terran/Protoss/Zerg in the game lobby

There's actually an important strategic difference: If an opponent picks Random race from the drop-down in the game lobby then you don't know which race they are using until one of your units comes into sight range of one of their units. Conversely, if either Terran or Protoss or Zerg is picked from the drop-down at random (with 1/3 probability each) then you know their race at the very start of the game. If you know their race then it cuts down the types of opponent strategies you need to prepare against or scout for (e.g. zerg rush, cannon rush, bunker rush) and may make it easier to decide whether to wall off, whether/where to place a bunker, what build order you want to use against that race, how early to send out a scouting worker etc, if you have not seen one of the opponent's units yet.

14. Add dependencies for BWAPI/bot-related libraries

I wasn't talking about the files installed by the Starcraft / Brood War installers/patchers - I was suggesting to add (and occasionally update) the set of BWAPI-related and bot-related library DLL or other kinds of library files that are common requirements by many bots, e.g. the BWAPI bridge and Qt DLLs that I mentioned, the client-bridge, gmp/libgmp/mpfr DLLs from sc-docker that I mentioned. And config files used by some bots like Aiur to configure where the correct locations of the read and write folders are (directories.txt and file_settings.txt files in the Starcraft program folder) to stop them updating files in the Starcraft folder itself. Aiur and other bots like that look for those kinds of files in the current working directory, which should normally be the Starcraft program folder. I expect it's easier for competition organizers and users that want to test their bots if a set of files was maintained rather than having to trouble-shoot to identify the missing files / try to find them online. As I said, I can add the files I have to a fork if you like. Note: for Java bots, I'm not sure about whether JAR files for BWMIRROR, JNIBWAPI, JBWAPI, BWAPI4J etc should be included in this set of files because I'm not sure whether they are supposed to be included within the bot's JAR, or must be in the bwapi-data/AI folder alongside the bot's JAR.

[davechurchill]

10. Will not be a priority for us as it will not be a feature in the AIIDE competition. I'd rather Rick spend as little time as possible on extra features outside the scope of AIIDE since we're running short on time getting the ladder up and running