Collect Images of TFT Units (for AI Training)

[Jarr0d161]

I played around with openCV and matchingTemplates. I'm able to track the items in the carousel (see attached file).

It took 300ms for my laptop to check for all possible items (54 or something) within one frame. It's because the algorithm compares the pictures of every single item with the cropped screenshot. I try to implement a method how to walk at the moment, but I don't have a working solution yet. Nevertheless - I can share the Code, if you want?

If we really decide to go with deep reinforcement learning, we won't need it, but it might be a workaround until then.

[Bonifatius94]

Damn. Good job! 😄😄😄 I'm currently a bit busy with other projects, but I'd like to see how you're progressing 😄

[Bonifatius94]

If we have a couple of pictures of each item / unit, it's quite easy to classify them. We can use CNNs with a dataset using augmentation methods (rotate images randomly, zoom in / zoom out, randomly crop and classify only parts of the image, etc) to create more data for training 😉

[Jarr0d161]

I don't think NN is the best approach here, since we don't want to create an AI that plays the game, but using ML to help us with some issue, caused by the time delay on Twitch. The pictures of the items in the carousel are not dynamic. Neither the content nor the size of the item boxes is changing, so I thought using those as the fix point is much easier. If we want to have a unit tracking option, we will need a lot of labeled frames and this is a lot of work - something I wanted to avoid. If you really want to go this way, we should assimilate the twitich community for labeling :D

I'm playing around with an implemented tracker algorithm in openCV and this works very good on static objects within a video. My suggested work flow for the item detection is:

1. Read out all Items in the carousel with matchingTemplate.
2. Community chose the item they want to have.
3. Again using matchingTemplate to find the start point for the tracker.
4. Using the openCV tracker for the chosen item AND for your avater (the small thing that is you :)). So you have the coordinates of those 2 objects every frame.
5. Now one can calculate the circular speed of the item (some easy circle function) and you can also calculate the speed of the avatar once it starts walking. Together with some other math, one can easily calculate the intersection of those 2 functions.

I have recorded a video while I was palying TFT to find patterns I can use as fix points. For example when to start tracking items and avatar, since we have this zooming at the begin of the carousel. But I think we could use the number of rounds (1-1, 1-2 ...) and the timer close to that, to handle some timing issues. For this we should track the screen frequently - at the moment the algorithm is just capturing, if a command is send. Since Maxim has the Aegis, we could start capturing all the time - i guess :D.

[Bonifatius94]

I agree. I didn't mean to use AI training for detecting moving TFT units / items in the carousel. That could be actually challenging to collect the training data, etc. I just wanted to classify the static unit / item pictures in the shop (which can be achieved with other techniques as well). Maybe the issue title was a bit misleading 😅😅😅

What about streaming the TFT screen augmented with moving rectangles around the carousel units? We can number each rectangle such that the chat can vote for a unit by number.

For this, we'd need to drop the framerate a bit and do some real-time OpenCV stuff, but it should be 100% possible. Idk how difficult it is to layer GUIs on top of each other. I'm not familiar with the capabilities of common streaming software.

In OpenCV, the rectangles are basically free if we apply some edge detection / color filter. Maybe we can pre-compute where the units will be on the screen by a circular function as you've already mentioned such that we don't need to augment each and every frame.

I'm sure we'll figure something out that's not too much effort 😂

[Jarr0d161]

I'm able to track the avatar using the healthbar, since it's green and your own healthbar is always in the forground - some interesting things we can play with. And it also works with the matching templates and tracking of a chosen item. Next steps will be the movement - I have already some ideas ... maybe I find some time later. This would be a big improvement and I think Maxim will give it a try - so he can test the whole "new" coding.

[Bonifatius94]

Nice, I'm really looking forward to Maxim testing the refactored version 👌🏻 Currently there's still an issue with the chatbot threading as background task. I'll need to figure out why it's failing and fix it. Once we have the confidence through test coverage, it should be easy though 😄

Maybe we should add some chat commands for the new TFT features of Set 7.0 😅 It's rather simple stuff like this new item shop round or the re-roll of augments. We basically just need to include the positions and implement the command with a couple of lines, no big deal. Should be within 1-2 hours 😂