search

Jarr0d161 / TeamFightChaticts

TwitchChat spielt TFT über Chatbot
Apache License 2.0
18 stars 1 forks source link

Replace Tesseract #5

Open Bonifatius94 opened 2 years ago

ghost commented 2 years ago

Discussed Ideas on this:

For number recognition, you can pretty safely use a low level classification algorithm (regularized logical regression NN) that should run on any computer.

Tesseract is designed for print, not handwriting - ergo the pre-trained models have seen print, which is why it works well here. If you really only want to read out the numbers and with as little computational effort as possible. Then I would neither touch Tesseract nor TensorFlow (neural networks), but work with pixel masks and regression methods (opencv, scikit-learn).

I just wanted to use TensorFlow and would do the forward pass with the pre-trained model weights in NumPy (doesn't have to be much more complex than MNIST with some dot products and ReLU / Sigmoid activations for a start). With my suggestion we would not need to add any new dependencies to our project, because we already have NumPy. But do what you think. I'm not angry with anyone if you have a better idea. Then maybe I learn something ^^

Jarr0d161 commented 2 years ago

Met Maxim today and told him, that you guys contribute a lot and that you bring this project to a next level - he was happy to hear. He had 2 ideas:

  1. Adding a voting system for a certain combo/set (like Assassins) and auto-buy it, the moment the unit appears in the shop. Here we need a little more complex OCR-pipeline. If we still wanna get rid of Tesseract, we could use - as @Bonifatius94 already mentioned - MNIST and maybe some "Fonts" generated Texts to train some easy NN. A wordlist, with possible Champions and classes will help us, finding always the correct unit, so it's fine, if the character error rate (CER) of the OCR-pipeline isn't too good. Additionally we need a Overlay, so people see the chosen combo. We discussed this to handle the time delay in the stream and giving the community the chance to play more competitive.

  2. We should open new issue for that, but still I want to mention it here. Finding the items in the carousel and people can vote for the item, which should be picked (listing also the units will be too much of work in terms of labeling ... I guess) and the avatar is moving indipendendly to the unit with this item, without accidentally running into a Champion with a wrong Item.

ghost commented 2 years ago

Okay, the first feature will than influence this Issue here. I guess I'll need some help to implement that. So using MNIST and some easy NN to recognize numbers is not that hard. I'm just not that sure how we should handle the rest.

How i would solve the problem: Iterate over the champions in the buy section and process each image by champion names / class names. The classification should then return the champion / set which matches the image. (So just OCR pipeline using nn which is trained with texts in different fonts) Valid classes are then a combination of champion name + set for all available TFT champions (champion / class wordlist)? If this champion is in the combo or the set matches, it shall be bought. If I'm completely wrong here please correct me :D

So should all recognition (Number & texts) be handled by a single nn or how would you approach that? Any good datasets you have in mind for training the OCR?

Bonifatius94 commented 2 years ago

2. We should open new issue for that, but still I want to mention it here.

Already opened the issue for you 😉

Bonifatius94 commented 2 years ago

Okay, the first feature will then influence this Issue here. I guess I'll need some help to implement that. So using MNIST and some easy NN to recognize numbers is not that hard. I'm just not that sure how we should handle the rest.

I've had all kinds of AI projects at university for the last 1.5 years (last one was a self-driving car software including some traffic light detection NN, etc). I'm very confident to solve this with ease. TFT is a very constrained environment because units always look the same. The only difficulty is when a new TFT set appears because then the look of the units changes. But it's not too critical either as we would just need to train a recognition logic with new data. This is more about the data really, the training and AI part is rather easy 😄

Bonifatius94 commented 2 years ago

So should all recognition (Number & texts) be handled by a single nn or how would you approach that?

Depends, but in this case I would use different NNs for each classification task. One for classifying units in the shop, one for reading texts / digits (and maybe one for detecting where specific objects are located at such that it can cut image patches and forward it to the classification networks to identify what it's showing). But more importantly, we also need some AI to play intelligently (because it's an auto-pilot), so we definitely need some kind of reinforcement learning as well (which I'm gladly very familiar to) 😄