nomis51
commented
3 years ago I finally took the AI solution, since Ctrl + C is too slow and invasive (I need to give the game a small delay before Ctrl + C, which alone is not too bad, but when you have 5 x 12 squares to scan in a window, it's a lot of delays) and image recognition is not accurate enough for more complex scans, like stack size or item links.
Might be able to have a working beta version soon with the AI able to read and understand the content of the trade window, once I resolve some technical issues.
One thing I hate when trading is to always have poe.ninja and a calculator opened in case I need to convert let's say exalted value to chaos value or deal with buyers who don't put full stacks of currency in the trade window. This is the general idea : Reddit post. Another interesting use case, would be to verify item links, like ensure it's a 6L and not a 5L / 6S to prevent scams.
I don't know yet if it's going to be a feature in the app, but i'm looking at it, because 1) it's fun to try and learn new stuff :) 2) it's a feature that should be in the game already so, it would be nice to have it.
Possible solutions
I was thinking about 3 possible solutions : ### AI #### Pros : (When trained properly) - Very accurate - Fast - Not intrusive (AI just look at the screen without in inputs, like human does) #### Cons : - It's a lot of work just to make it work (And it's not even good) - Even if accurate enough, it's never going to be 100% accuracy ### Image Recognition / Text recognition : #### Pros : - Fast - Doesn't need a lot of work (except if training for OCR) - Easy to use - Accurate (in most situations) - Not intrusive (Match algorithm just look at a picture of the screen, no inputs involved) #### Cons : - Not accurate enough in some situations (e.g. very detailed matching) - Can be heavy on CPU if multiple match needs to be done - Even if accurate enough, it's never going to be 100% accuracy ### Automated in-game Ctrl + C : #### Pros : - Easy to use - Very accurate - Rely on real data, not inffered data with a % chance of being wrong #### Cons : - Need to grab mouse and keyboard to move the mouse over each items and perform a Ctrl + C on it (Player can't interact with the game while it's being done) - Very slow compared to other solutions (not extremely slow, just a second or two vs couple milliseconds both other solutions) So far, I've tried all three solution : 1) Quite accurate in most tests, but need more training, and there's a lot of code. Edit: After training different setup of CNNs, I think I got something pretty good. Models are now able to predict with very good accuracy. AI able to find the right currency type (e.g. it's a chaos orb on the image) ~98% of the time (Got difficulties with "empty") and to find the right currency stack size (e.g. it's a stack of 10 currency on the image) ~99% of the time (Again difficulties with "empty") and to find the right amount of links on an item near 100% of the time. It's going well so far! 2) Very accurate to read currency type (e.g. it's a chaos orb), but not really when trying to read the stack size (e.g. **3** chaos). Very easy solution, but not accurate enough for now. 3) Very accurate and it actually work pretty well! But, it grabs player's mouse and keyboard... Here is a showcase of the solution, using poe.ninja prices to convert to chaos : 