update the scanner to scan according to game scaling and make it a lot more accurate

[meepen]

This now uses confidence to base its results on the most confident guess per slot in the inventory instead of whatever it was doing before.

[meepen]

note: image recognition only works at 1080p or above because of the size of the images at lower qualities not being detected as confident matches.

[4rtzel]

I pulled your changes and tested them a little bit. As I understand, the main idea here is that we can calculate the position of each slot in the inventory and based on that, determine the best matching image for each slot?

I see that you've also removed the scanning window parameter and now calculate it based on window info.

Overall, I like the changes, since it improves the searching algorithm. My only concern is that it feels slower than before. I'll try to look into it.

[4rtzel]

I can confirm that for me, the scan time increased from 2.8 seconds to 9.0 seconds. That's a significant bump.

[4rtzel]

It seems that the main offender is mask=mask parameter in cv2.matchTemplate function. By using it, the average time spent in cv2.matchTemplate increased from 33ms to 140ms.

@meepen, how important is this parameter for search accuracy? If we're going to select one match for each item slot anyway, could we omit it for performance sake without loosing in accuracy?

[meepen]

We can completely remove the mask argument as it's not strictly necessary. I will do that now!

[4rtzel]

Testing on windowed 1920x1080. Unfortunately, the changes don't work that well on this setup. I'm getting lots of false positive and some items are not recognized with it. Here is the screenshot: .

[meepen]

I am also getting those issues, I will look into it more.

[meepen]

So it seems that I had a few calculations wrong (image size is actually - 1 because there is a 1px border between each section) As well as that, I have used masks to make it basically perfect, however the slowdown does still occur. I am going to use threads (hopefully that's alright) and it will be right as day and should potentially be even faster than before.

[4rtzel]

I'm not really a Python developer, but from my understanding of how GIL works internally, you probably won't be able to gain much performance out of multithreading.

I think there might be some problem with image scaling for resolution != 1440p that causes low confidence results.

[meepen]

Does this commit solve the things you and I were having? I am running at 1080p and it seems to work a lot better.

[meepen]

Further testing shows it seems to work a lot better at smaller resolutions too.

[4rtzel]

It does seem to work better on lower resolutions. One thing to notice is that there seems to be an accumulated error for slot positions. You can see it here, where the highlights is a little bit off:

Another thing is that it doesn't detect some items in 1440p:

First two, Brine King-Touched and Shakari-Touched are not detected.

Lastly, with multithreading pressing the scan button consumes all of your CPUs for a couple of seconds which causes freezes for all programs.

[meepen]

can you provide a full screenshot of the inventory so i can test it? in my 2560x1440 game it detects everything properly

[4rtzel]

https://i.imgur.com/jGktlTk.jpg 2560x1440

[meepen]

with that image i get the following:

[meepen]

I've fixed the other issues you have mentioned, I do not know if it's something we have different that causes this to happen.

[meepen]

Oh! The confidence is set to 0.75 for me.

[4rtzel]

@meepen, my plan is release v0.2.0 with https://github.com/4rtzel/poe-archnemesis-scanner/pull/37 changes + keyboard changes. Then I'll merge yours for v0.2.1 release. I want to allow people to get back to v0.2.0 if your new algorithm won't work on their setup out of the box.

I hope that's okay with you, and thanks for the contribution!

[meepen]

sounds fine! i will resolve the conflicts in the meantime

[meepen]

I believe this may be undercooked so I am going to revamp for a bit. Sorry for that :(