search

beuaaa / pywinauto_recorder

A record-replay tool to automate GUI via pywinauto
https://pywinauto-recorder.readthedocs.io/
MIT License
131 stars 24 forks source link

How about Add the functions of finding function points through images ? #29

Closed LRF520 closed 3 years ago

LRF520 commented 3 years ago

Hi,Beuaaa.How about Add the functions of finding function points through images ?,just like:

  1. Add the functions of finding function points through images ;
  2. Add the functions of image comparison, and output the percentage result of the comparison.
beuaaa commented 3 years ago

Hi @LRF520, Do you mean a find method like in SikuliX or Lackey?

LRF520 commented 3 years ago

tools like SikuliX are so alone , because they only have image functions,in the actual and complex working environment, this single method is difficult to work alone.

In real life, most of the time, we hope to guide the next operation through image matching technology, or judge whether the video picture meets the expectations. Therefore, the need to integrate image recognition technology into the recorder is huge. for instance:

  1. When we play the game, we hope that the recorder can recognize and record different game character images so that the game can run by itself next time;

  2. We have synthesized a long video, and we will prepare some images for comparison in advance. When playing these videos over and over again, if the recorder can use these pre prepared images to compare with the real-time video picture, a percentage result will be automatically calculated based on the comparison of each image, and we can finally roughly judge whether the quality of those videos is reliable.

beuaaa commented 3 years ago

Generally speaking, sometimes it is indeed difficult to automate something perfectly with a single automation method. When it is necessary, it is possible to enrich a script generated by Pywinauto Recorder with functions from other libraries (Lackey, OpenCV, OCR, ...)

About your first example: Lackey can solve this kind of need very well.

About your second example: Comparing a video recorded by a script with a reference video is one of my needs. I haven't looked for it yet but I imagine that this kind of algorithm exists and that it is possible to find an open source implementation. In this case I don't see the need to integrate this functionality in Pywinauto Recorder because you can use the existing libraries with Pywinauto Recorder very well.

Or maybe I misunderstood?