asweigart / pyautogui

A cross-platform GUI automation Python module for human beings. Used to programmatically control the mouse & keyboard.
BSD 3-Clause "New" or "Revised" License
10.45k stars 1.26k forks source link

Pyautogui locateOnScreen() not working in full screen #648

Open DarkPrince27 opened 2 years ago

DarkPrince27 commented 2 years ago

Hey guys I'm new to python and I have spent roughly 30 hours trying to figure out how to fix this issue. Sorry that this is so long I tried giving as much info about what I tried already.

I am searching for a cat walking in range of my standing character in a full screen game using pyautogui locateOnScreen(). The problem is that the image is found when I have the shell or command prompt window running the script open in front of my game but if I start the script and focus the game screen immediately, the image is not found at all.

I created a new script with only the cat finding part since the rest of the main script is only keyboard input commands to run to the cat and then stand still.

import pyautogui
import time
import win32api
import keyboard  # in my main script I use hotkeys and keyboard seemed to be the simplest route

def main():
    time.sleep(3)
    print('ok ready')

    while not keyboard.is_pressed('c'):
        if pyautogui.locateOnScreen('catplant.png', region=(260, 450, 300, 250), confidence=0.4) is not None:
            print('cat found')
            keyboard.press('o') # pushes button to start interacting with the cat when found
        else:
            print('cat not found')
        time.sleep(0.1)

if __name__ == '__main__':
    main()

note: the region is 100% correct + extra range since i can find the cat with the command window open and I used win32api.SetCursorPos and a for loop to outline the range and then tested to see when the cat should be in range. Also the confidence has to be 40-50 % as the background seems to be too similar to the cat image and the cat has different animations while walking.

The thing that infuriates me the most that sometimes when only the print('cat found') line is there , it will show that the cat was found, though still inconsistent. As soon as I add any command, mouse click or keyboard input after the cat is found, the script does not find the image at all besides with a command window open. Does not work when print is replaced with any other command either. I tried adding delays as I know the function can be slow. Didn't work.

I've checked that the active window is the game using code just to make sure and the results are still the same.

Also when I run the script and the cat is not found as soon as I alt+tab the cat is found if in range and the cat will also sometimes still be found despite the window now hiding the cat. This does however not work when i hide the cat with windows in other instances like in a picture of the cat in paint or the starting with command prompt window in foreground for example.

The problem is that its almost as if the function cant find the image until the shell pops up in the foreground (during the search) and then all of a sudden everything works perfectly. But I cant keep the shell up while the script is happening as the game is fullscreen and the script sends keyboard inputs.

I might not be very experienced but I have a feeling it might be an issue with pyautogui. If I can't find a solution I'll have to resort to OpenCV which seems quite complicated and I would not prefer going into that for now. Thank you for taking your time to read and potentially try to solve this problem for me. This is my first post on this website.