Update League of Legends V13

[Neyzn]

Hello, could you update this code? I tried to launch the scrapping and no overlay appears, I do not know if it comes from me or not, league of legend launches with the replay but nothing happens in the terminal and no overlay.

[MiscellaneousStuff]

Hello there. If you’re using this with tlol-scraper, I’ve stripped the normal overlay that the script displays on League to optimise it for scraping performance. The scrapers offsets haven’t been updated since towards the end of season 12 hence it’s not currently working. I’ll probably update it for next patch, 13.3 (next Wed) instead of now as new patches require new offsets so it would quickly become redundant.

[Neyzn]

Ok thanks, and to launch the ia how do you do it?

[MiscellaneousStuff]

This video still contains the correct instructions. They just won’t work atm because the most recent tlol-scraper release is out of date.

https://m.youtube.com/watch?v=Mz7NbIgJqsc

[Neyzn]

Yes, in the video you show how to use tlol-scraper but you don't show how to make the ia play and train it with .db files

[MiscellaneousStuff]

To train with the .db files is very dependent on what you want to do. If you only want to train on the .db files you don’t need tlol-py. You can use existing datasets from tlol. I would recommend the Jinx833ML dataset as it contains observations and actions so you can actually train the model to play the game.

If you want to train a model and be able to use a model to play the game, this is a lot harder. You need to install zerorpc, and run the release version of tlol-scraper (optimised version) with this script running for LViewLoL. This will setup an RPC server for PyTorch to send commands to League of Legends so it can play the game.

To train a model, you can use the Jinx833ML dataset from above (or only use 10% of the dataset to start with), and train it using python -m tlol.bin.train —db_dir path_to_replay_dbs/ You then need to run python -m tlol.bin.rpc —model_path your_model.pt

However I will say that this project is a research project so don’t expect the model to be very good at all. It would take a lot of domain specific knowledge about MOBAs / League of Legends to get the right features from that dataset and make the right model to play the game to even a playable level.

This blog post explains the complexity of it

[Neyzn]

ok, and have you considered teaching him the basic rules of the game, and then have him play against himself in 5vs5 (I don't know how it would be possible to do that, but it's potentially possible) so that he can train, a bit like the Rocket League A I?

[MiscellaneousStuff]

What Rocket League AI project are you referring to?

[Neyzn]

I'm talking about the AI Ragnarok, I don't know if you understand French, but you can watch this video, it talks about it and explains a bit about how it works. https://www.youtube.com/watch?v=HCYwWudJxho

Basically, it says that the AI runs on servers and launches thousands of games, and in one month a bot can train for 273 months. The AI plays against itself. The person who created this AI made a reward system that he modified a bit, because for example in RL, you shouldn't take a boost to take a boost or else the AI would take all the boosts on the map without trying to score. The problem with that is that you need money to rent/buy servers.

[MiscellaneousStuff]

Ok I think I understand what you are asking for now. What you want is an OpenAI Gym style environment for League of Legends so you can train an agent to play League of Legends using reinforcement learning (e.g. PPO like in the Rocket League RLGym example).

So for example, what you would want is a python-like environment where you could do:

import tlol.gym as leaguegym

env = leaguegym.make()

while True:
    obs = env.reset()
    done = False

    while not done:
      #Here we sample a random action. If you have an agent, you would get an action from it here.
      action = env.action_space.sample() 

      next_obs, reward, done, gameinfo = env.step(action)

      obs = next_obs

Where this game loop would you let control a champion in game and your agent could learn to play the game?

In terms of playing 100s or 1000s of games, technically yes you could do the equivalent of this but you would need a large amount of hardware to do it. Without getting help from Riot, what you could do is launch a client (so you would need to be able to launch multiple game clients at once) for each agent which is going to play the game. If you want 5v5 games, you could setup custom games and have an agent controlling each player in the game. This would require extending the tlol-py module with similar code to pylol.

If you could convince me that there is enough interest in a project like this, I will strongly consider extending tlol for the functionality you are asking for.

Edit: I've posted an open discussion for what you are suggesting on Reddit. If there is significant interest, I will consider going forward with it. Reddit post.

[Neyzn]

I think the project can be very good, if AIs are not banned from league of legend. Or to make business, but that's not my goal. Do you know if AI's are banned ? on the other hand I didn't understand the code, I've been interested in AI for a short time so I don't understand the code.

[MiscellaneousStuff]

Good question, basically yes you can’t make bots if your goal is to help players cheat at the game. However the goal of this project is research and many people have published research creating League AIs (see more here) so this project should be fine in theory.

The code above basically breaks down to a simple loop where you do the following:

1. an agent (so in League the champion that the player is playing, or in Rocket League the car that you are driving) takes an action (so in League this could be auto attacking someone and in Rocket league this could be steering left)
2. After you take the action, you then see what the game looks like afterwards (e.g. in League after you auto attack someone, you would expect their HP to go lower). Along with that, you also get a reward. So after auto attacking someone you would probably get a positive reward of like +0.5 or whatever it is. And this keeps happening until the game stops or the agent wins.

[Neyzn]

ah ok, it's perfect, we just have to make sure that the agent doesn't hit someone just for the reward, it has to be useful to fight, and I have no idea how to do that. and do you plan to do it or do you still doubt ?

[MiscellaneousStuff]

Yes I plan to do it now. You can follow along better here. https://discord.gg/G9nmukGn

[Neyzn]

Do you think it would be possible to train it quickly and on its own PC ? Because it costs a lot to buy servers.

[MiscellaneousStuff]

The way it would work would be to run multiple league clients at once with an agent controlling each player in real time

[Neyzn]

so, on a single PC?

[MiscellaneousStuff]

Yes, it's possible to do on a single PC. You would need a decent PC, but nothing over the top just something that can run multiple League clients at once. However, you can reduce the resource usage of League by turning down textures, shadows, anti-aliasing, resolution, etc.

[Neyzn]

Ok, then I wait for the news of the project on the discord ?

[MiscellaneousStuff]

Yeah that’s a good idea. Progress is coming along very quickly at the moment. I have already ported all of pylol (very similar related project for an older version of League) and the only bug remaining is some synchronisation with RPC (making sure OpenAI Gym is synched correctly with League). You should expect to be able to train RL agents using an OpenAI Gym environment as early as today or tomorrow.

[Neyzn]

Okay, perfect. I would so love to know how to make AIs :/ I'd have to read up a little more

[Neyzn]

Is the AI going to learn by itself the goal of the game? When do you have to take drake, herald, nashor. When do you have to farm or hit your opponent, or do you have to modify your code to do that?

[MiscellaneousStuff]

Reinforcement Learning-Based Macro Decision Making

That's a good point. When Deepmind and OpenAI created their AlphaStar and OpenAI Five bots, respectively, they both started off with mini-games for the RL agent to learn basic things about the game. When the agent could reliably learn to perform well in these mini-games, they gradually scaled up the complexity of the tasks. In the context of League of Legends, this could start with having a bot spawn in the middle of a practice tool game, and learning to last hit minions. Then once we can reliably train the bot to do that, we put it in a lane against an enemy bot and we will have it learn to last hit and attack the enemy bot. In terms of when it should do certain things, from an RL perspective the bot learns more "macroscopic" decision making by trying different things and finding out what gives it the highest reward for that situation. So if trying to shove a wave in and take dragon gives it more reward, it would gradually learn to do this over time, that is how RL agents learn macroscopic decision making.

Supervised Learning-Based Macro Decision Making

Another alternative to this approach could be to combine supervised learning and reinforcement learning. You could separate the bot learning macro (where to be on the map), with the bot learning micro (what actions to perform in it's current position).

You could teach the bot macroscopic decision making using supervised learning. From a low-level perspective, you could breakdown the League of Legends map into a 28x28 grid (or w/e size) and treat where an agent should be on the map, almost as a game of chess. You could then train the bot to learn macro by comparing where it predicts it should be on the map (28x28 grid position) and comparing it to high elo players decisions in similar situations.

You would call this the "global intent region", which is one of the 28*28 := 784 map positions. Then for the low-level decision making, you could use reinforcement learning to learn how to play teamfights, laning and other situations which have higher variance. This part is where tlol-rl becomes useful. Alternatively, you could also train the agent how to play "micro" situations using the same method as "macro", by breaking down the individual global intent region, into a 10x10 grid of "local intent regions" and then training a supervised learning model to learn how high elo players would have played the same situation.

Detailed Analysis of Existing Methods

The method above of using global intent regions and local intent regions along with supervised learning was used by TencentAI to train a bot which achieved pro player level performance on Honor of Kings (very similar mobile game to League). I have analysed and broken down their original research paper here which goes into far more depth about the specifics of this method.

However, the idea of controlling the micro behaviour of the bot using reinforcement learning is also adapted from other Chinese MOBA AI papers with some adaptations.

[Neyzn]

Wow, finally it does not look so simple, but it is even more interesting