Make bots better

[wrmacrae]

This is an enhancement we can survey for interest on Survey 2. Some feedback on Survey 1 and elsewhere has mentioned that the Bot AI isn't strong enough to feel like good practice, so this is a potential area for investment.

[tooomm]

Found out about this: https://github.com/madrury/mtg-draftbot

[madrury]

@Senryoku @tooomm @wrmacrae Y'all want some help with this? I'm the dev of the project linked above. It'd be fun to contribute here.

[Senryoku]

@madrury For sure! I only skimmed the readme of your project, but something like archetypes-aware bots would help a lot. I see data for M19 and M20, they look like the generated default weights mentioned in the readme; did you managed to find enough data to train your model?

In any case, improvement to the bots are more than welcome; I'm sure a lot of users would want to train with better opponents.

[madrury]

@Senryoku I never did follow through on finding a data source, I got a new job and that occupied a lot of my time for a while. So I mostly just got the project to the point where the models could reconstruct my personal archetype/pick weights from simulated draft picks, which is a good proof of concept.

We certainly could use the approach with pre-defined archetype weights like I used to simulate data, that would lead to reasonable, but exploitable, behavior.

Do y'all have applicable data here we could use for training? If you're collecting pick-by-pick data with:

• What cards the drafter has in their pile.
• What cards are available to pick.
• What card is chosen.

That's what would drive the model.

[Senryoku]

I tried gathering data for a time, but using DynamoDB for that was a very poor choice :D If we find a free and convenient way to store player picks, I could re-enable that. Not sure if we'd get enough data to be relevant, but it's always something we can try.

[tooomm]

There is also this one: https://github.com/RyanSaxe/MagicDraftBot

[ruler501]

Would there be interest in incorporating the draftbots from CubeCobra? I'm the primary developer on them and they're getting pretty constant improvements and testing on CubeCobra.

[Senryoku]

I'm not familiar with CubeCobra's bots, but that's probably an improvement over our current ones :) What's their overall strategy? How difficult would the integration be?

[ruler501]

I have started writing a series of articles on CubeCobra about how the bots were developed and how they work. Parts 1 and 2 are up on the site and 3 is done and in the queue to get released soon. The first 4 parts will cover drafting up to what is currently implemented in the dev branch. Part 5 will go over deckbuilding (which we're pretty bad at), and 6 will be concepts and in progress work.

As for integration the main hurdle will be to get the card data from CubeCobra, specifically need the elo and embeddings that aren't in the Scryfall data. We should be able to setup a protected endpoint for that. That would also be useful for a lot of the local experiments and testing I do. There are full javascript and C++ implementations already being maintained (as well as futhark and CUDA, but those aren't really useful except for large scale simulation).

I would greatly appreciate it if you could track drafts so that for every human pick I can reconstruct the DrafterState(https://github.com/ruler501/CubeCobra/blob/b715bce4841304ef9e0c285b1e377176192d2b66/src/proptypes/DraftbotPropTypes.js#L11 and some of the fields would be optional) as well as which card the human picked. That will help me do experiments and improve the draftbots(can see some of the work at https://github.com/ruler501/CubeCobraRecommender/tree/parsed-cards-code2seq and https://github.com/ruler501/DraftBotOptimization the second has some local changes to update for some small changes and optimizations locally I haven't pushed up yet).

[ruler501]

I guess a short summary of the bots is that they are a 3 tier process defined by this equation(might need to disable dark mode to read it since it has a transparent background and black text).

So at the inner level it consults several oracles which are specialized to analyze different aspects of how good a card/land combination is. It uses a linear combination of the value each oracle returns with the weights determined by the oracle and the pack/pick number. Our current oracles look at Rating(roughly how much more likely it is to be picked than the average card), synergy(determined by our machine learning model, is roughly how well cards go together), the cards already picked, whew and cards seen so far in the draft.

The second level gives the bots their power. They are constantly considering what their final deck will look like. Specifically they always have a plan for what the color combination of each of the 17 lands in their final deck will be. This is done by having the bot find the combination of lands that maximizes the score from the inner level. The oracles use the lands to calculate what the probability they will be able to play each card on curve is which they use to weigh the value it gives. Currently I just have it use a fastest ascent hill-climbing algorithm to do this from a predetermined starting point with the transitions defined as swapping one basic for a different kind of basic. The score from the inner level at the maximum is what gets passed on from here.

The third level just selects the card in the pack that the second level gave the highest score to.

[Senryoku]

Thanks for the detailed answer (and articles)!

I'm storing draft logs like this one: 2021_02_22T06_25_59_021Z.zip, it's not the most elegant file format but it should holds everything you need to reconstruct your DrafterState. If you're interested I can find a way to transfer everything I've gathered so far to you (I can also pre-process it beforehand if you want, filter bot picks or strip the carddata property for example).

Would you be willing to start a PR for the integration? I honestly have no idea how to organize such a collaboration. Are the "embeddings" you mention related to the adjacency matrix for the recommender?