moderators can configure pool- and project-formation algorithms using the CLI

[jeffreywescott]

Overview

As per https://github.com/LearnersGuild/game/issues/418, the moderator needs a way to configure the pool- and project-formation algorithms. Each of these algorithms has a different relationship to Elo rankings.

The suggestion from game mechanics was two configuration "files" to be passed to cycle init and cycle launch commands. However, uploading files to echo would be difficult to support in a clean way. Engineering suggested the alternative of configuration URLs rather than files.

OND: it seems to me that one configuration file with different sections would be superior, and perhaps a moderator-only command to update the configuration makes the most sense.

Command / Arguments

View full configuration

moderator only

@moderator > /config
@echo      > Configuration
             -------------
             pools:
               - id: green
                 elo_band: 0..1000
                 advanced_players:
                   - @bundacia
                   - @shereefb
               - id: blue
                 elo_band: 1001..2500
                 advanced_players:
                   - @jeffreywescott
                   - @prattsj
             project_limits:
               elo_0: 1
               elo_1100: 2
               elo_1300: 3
               elo_1500: 4

Configure voting pools

@moderator > /config add --pools id:purple elo_band:2001..9999 advanced_players:[@tannerwelsh, @jeffreywescott]
@echo      > Config added.
@moderator > /config update --pools id:blue elo_band:1001..2000 advanced_players:[@jrob, @deadlyicon]
@echo      > Config updated.
@moderator > /config delete --pools id:purple
@echo      > Config deleted.
@moderator > /config get --pools
@echo      > pools:
             - id: green
               elo_band: 0..1000
               advanced_players:
                 - @bundacia
                 - @shereefb
             - id: blue
               elo_band: 1001..2000
               advanced_players:
                 - @jrob
                 - @deadlyicon

Set concurrent projects limit

@moderator > /config add --project_limits elo_1200:2
@echo      > Config added.
@moderator > /config update --project_limits elo_1500:5
@echo      > Config updated.
@moderator > /config delete --project_limits elo_1100
@echo      > Config deleted.
@moderator > /config get --project_limits
@echo      > project_limits:
               elo_0: 1
               elo_1200: 2
               elo_1300: 3
               elo_1500: 5

Help messages

Top-level command

@moderator > /config --help
@echo      > /config - game settings

             Usage:
                 /config [options] <command>

             Commands:
                 get            - View a game setting

                 add            - Add a game setting

                 update         - Change a game setting

                 delete         - Remove a game setting

             Options:
                 --help, -h            print usage information
                 --pools, -p           configure voting pools
                 --project_limits, -l  configure concurrent project limits

             Examples:
                 # Add a new voting pool
                 /config add --pools id:purple elo_band:2001..9999 advanced_players:[@tannerwelsh, @jeffreywescott] 

                 # Update the concurrent project limits for players with Elo >= 1500
                 /config update --project_limits elo_1500:5

Example subcommand

@moderator > /config add --help
@echo      > /config add - game settings

             Usage:
                 /config add <options> <data>

             Options:
                 --help, -h            print usage information
                 --pools, -p           add a voting pools
                 --project_limits, -l  add a concurrent project limits

Configuration Format

Arguments to command. No file upload.

Feedback to Moderator

• Successful commands respond with simple success message.
• Invalid/unparseable respond with standard error message, directing to --help
• Moderator can view all config settings at any time with /config or /config get [opts]

  Player Visibility of Configuration

Not visible. Voting pools and concurrent project limits may be made visible through other means. /config command is only accessible to a moderator.

[jeffreywescott]

Open UX Issues

• How should the moderator re-configure the game? What command + arguments should be used?
• Will there be configuration "files" or "URLs"? If so, what is the file format for the configuration? YAML?
• What feedback should be given to the moderator from the command once its issued?
• Is the configuration information public / private? In other words, can non-moderators "view" it somehow?

[jeffreywescott]

@tannerwelsh LMK if this works as a starting-point for UX work. Ideally, I'd like to provide something like this for everything that is "ready for UX". LMK if something else would work better.

[jeffreywescott]

Suggestions from Game Mechanics:

Pool configuration

The moderator passes a pool threshold config file to /cycle init which defines:

1. The number of different pools
2. The Elo thresholds between pools
3. Advanced players assigned to the pools

For example the config file could define:

Pool Elo min	Pool Elo max	Advanced Players
0	1000	@bluemihai, @jaredtron
1001	1100	@shereefb, @tanner
1101	9999	@carla, @needdra

Project configuration

Once votes are in the Moderator launches the cycle /cycle launch passing a team threshold config file that defines Elo bands which constrain the maximum number of concurrent projects a player can be on.

For example, the team threshold config file could dictate:

Elo Threshold	Max Concurrent Projects
1000	0
1100	1
1200	3
1500	4

This means that a player with an ELO rating less than 1000 cannot be assigned as an advanced player to a team.

A player with an ELO rating of 1150 can be an advanced player on upto 3 concurrent projects.

[tannerwelsh]

Proposed moderator UI/UX for configuration. Feedback please! \cc @LearnersGuild/software @shereefb

Tried to keep the subcommands straightforward and easy to type, remember, and parse.

[bundacia]

I would strongly prefer a command that just takes a URL param that points to a yaml file (we could even add param to /cycle init). I think that would be a lot simpler to use and much easier to implement. Trying to type this correctly on the first try would be a bit of a pain:

/config add --pools id:purple elo_band:2001..9999 advanced_players:[@tannerwelsh, @jeffreywescott]

Moderators could use a git gist to host the file which would allow them to reuse configs from cycle to cycle and even track changes.

[tannerwelsh]

Roger that @bundacia. I'll update the design with a URL input instead.

Starting to feel like doing any command-based UI design work is comically useless. I agree that pointing to a YAML file is a "better" way to get the data into the system than the design above, and it is also one more glaring hack. Looking forward to designing towards fewer hacks.

[bundacia]

Yeah, it's a huge hack. The hackishness is mitigated a little bit in this case by this being a moderator-only command, so at least we're not negatively affecting the UI for the players and it contributes toward this benefit:

As a game designer, I can iterate quickly on the game mechanics

Since this will effectively allow the moderator/game-design to tweak the team formation algorithm in real time.

[heyheyjp]

...sorry if i missed it, but why wouldn't we do this as a form in a web app (flex panel)? no matter what the moderator interface is, the system still has to be able to identify and process specific inputs, so it offers no less flexibility for the moderator than a config file or strictly command line interface would. and it would seem to be the clear winner in terms of usability.

[tannerwelsh]

Here's what a web form could look like:

Moderator runs /cycle config to slide out this panel:

[tannerwelsh]

And what a YAML-file URL provision interface would look like:

@moderator  > /config update https://gist.github.com/raw/apeionafdli132r98bae/cycle_config.yaml
@echo       > Cycle config updated! New config settings:
              pools:
               - id: green
                 elo_band: 0..1000
                 advanced_players:
                   - @bundacia
                   - @shereefb
               - id: blue
                 elo_band: 1001..2500
                 advanced_players:
                   - @jeffreywescott
                   - @prattsj
              project_limits:
               elo_0: 1
               elo_1100: 2
               elo_1300: 3
               elo_1500: 4

@moderator  > /config update https://gist.github.com/raw/apeionafdli132r98bae/INVALID_cycle_config.yaml
@echo       > Womp womp. Couldn't parse that config file. 
              Did you verify that it is a valid YAML format, and matches the schema requirements?

[tannerwelsh]

So, we have 3 options on the table, each with their pluses and minuses.

1. Command-first approach that is consistent with the other CLI features
2. Command to point to an external file with config settings in YAML format
3. A web form made visible as a slide-out panel by issuing /cycle config

Given that the primary stakeholder is one person, and there are two potential implementers, the questions that remain are:

1. @shereefb - which interface do you prefer from the standpoint of moderator and game mechanics?
2. @prattsj and @bundacia - which approaches would be least costly to implement and maintain?

[heyheyjp]

neither the web form nor config file approach is trivial from an implementation standpoint. and at a glance, the config file parsing approach is not significantly less expensive to implement than a gui form. there are far more things that could go wrong because of constraints in the ui being necessarily implicit with a command line + config file interface than with a web form w/ validated input fields. the command line + cli input also asks the end user to remember more, to manage storing the file in an accessible place (public, or else we have to deal with authorization depending on where it's stored) and to generally work much harder to provide the expected input. my strong gut sense is that a web form, with some iteration on @tannerwelsh's initial mockup (great start, btw!) is a better solution.

[tannerwelsh]

oh, and there is another option (cheaper and less usable in the long run) - these settings could live in a YAML file within the game repo itself, and @shereefb could just modify that file directly and submit PRs against the codebase.

[heyheyjp]

no. 3 is probably the most expensive in terms of implementation (by what i think is not a prohibitively greater amount than the config option), but i believe it'll yield the greatest value in the end both for the moderator in terms of ease of use and for engineering in terms of ease of support/maintenance. it also gives us more room for iteration on the ux. there's a limit to how complex a strictly CLI approach (no. 1) can get before it's not worth bothering with. i think we're already just about at that limit, so there's no room to make the interface able to handle more complex input. no. 2 saves us some time perhaps in initial implementation but asks a lot of the moderator. whether or not it's too much only @shereefb can say.

[heyheyjp]

@tannerwelsh, @shereefb: another thing to consider is that in a few weeks, @shereefb ain't gonna be the only moderator on the floor. i think we'll wanna raise the priority of moderator usability for both our sake and theirs. : )

[heyheyjp]

@tannerwelsh: thoughts on web form mockup...

[bundacia]

The web interface is nice, but I think it would definitely be more work than just fetching a file-based config, and it makes adding new config options a little harder (with a YAML file there are zero UI changes to add options). The YAML file allows us to iterate really fast on tweaking the algorithm which I think is a high priority for @shereefb in his game mechanics role.

[heyheyjp]

with a YAML file there are zero UI changes to add options

If the web UI were done reasonably well :), this should be true for it, too. see: survey UI. it could allow for fast iteration on the very same kinds of changes.

[tannerwelsh]

Ok so @shereefb I think we need some more clarity from you then to make this decision:

1. Do you expect moderators to (a) always need to configure these settings, or (b) is this just for your experimentation in the near-term?
2. If (a), then let's do the right thing (web form).
3. If (b), then let's do the easy thing (YAML form).

[shereefb]

My strong sense is that this just a temporary hack so that we can experiment with different ways of forming the pools. After a few weeks of experimenting (ideally before November cohort starts) this should all be automated.

I think we should optimize for quick, cheap implementation that will get thrown away soon and replaced by algorithm, and we should assume that myself or @tannerwelsh are the only two users for this interim step.

my 2 cents: changing a config file directly in the code base would be fine.

[heyheyjp]

thanks, @shereefb. if we're going with the config file-in-codebase-option, there's no work to be done here in game-cli. moving this issue to the game repo.

[heyheyjp]

Issue moved to LearnersGuild/game #432 via ZenHub