Use LocateOnScreen with Multiple Monitors

[JohnTravolski]

I have an extra monitor plugged into my laptop many times (but not all the time) and I recently noticed that the following code:

    import pyautogui
    change_intense_to_calm = pyautogui.locateOnScreen('Intense.png', confidence = 0.9)
    run_script_button_x, run_script_button_y = pyautogui.center(change_intense_to_calm)
    pyautogui.click(run_script_button_x, run_script_button_y)

doesn't work when the 'Intense.png" item appears on the second monitor! It does move the mouse to the position and click if it appears on the primary monitor (the laptop screen), but if the window containing that item is on the secondary monitor, it fails, giving me this error:

    Traceback (most recent call last):
      File "E:\test_folder\switch.py", line 18, in <module>
        run_script_button_x, run_script_button_y = pyautogui.center(change_intense_to_calm)
      File "C:\Program Files\Python\Python37\lib\site-packages\pyscreeze\__init__.py", line 407, in center
        return (coords[0] + int(coords[2] / 2), coords[1] + int(coords[3] / 2))
    TypeError: 'NoneType' object is not subscriptable

presumably because it's not finding it.

How can I modify my code so that it will find the item and click it regardless of which monitor the window containing it is located on?

[JohnTravolski]

Does anybody understand why this doesn't work?

[jhuels]

Same issue here. I assume pyautogui only searches the monitor starting at 0, 0

[JohnTravolski]

Does anybody know of a solution? I'm dying to have this fixed. It would be so useful if it just worked.

[pascallo]

If still relevant for someone on windows:

In my opinion the issue is, that the current version of pyscreeze utilizing ImageGrab (Pillow) on windows only uses single-screen grab.

A dirty quick fix in pyscreeze could be:

• enable allscreen grabbing: In file: pyscreeze/__init_\.py, function: def _screenshot_win32(imageFilename=None, region=None): change im = ImageGrab.grab() to im = ImageGrab.grab(all_screens= True)

• handle new introduced negative coordinates due to multiple monitor: In file: pyscreeze/__init__.py, function: def locateOnScreen(image, minSearchTime=0, **kwargs): behind retVal = locate(image, screenshotIm, **kwargs) add

  if retVal and sys.platform == 'win32':
  # get the lowest x and y coordinate of the monitor setup
  monitors = win32api.EnumDisplayMonitors()
  x_min = min([mon[2][0] for mon in monitors])
  y_min = min([mon[2][1] for mon in monitors])
  # add negative offset due to multi monitor
  retVal = Box(left=retVal[0] + x_min, top=retVal[1] + y_min, width=retVal[2], height=retVal[3])

• don't forget to add the import win32api In file: pyscreeze/__init__.py,:

  
  if sys.platform == 'win32': # TODO - Pillow now supports ImageGrab on macOS.
  import win32api # used for multi-monitor fix
  from PIL import ImageGrab

[hapham117]

If still relevant for someone on windows:

In my opinion the issue is, that the current version of pyscreeze utilizing ImageGrab (Pillow) on windows only uses single-screen grab.

A dirty quick fix in pyscreeze could be:

• enable all_screen grabbing: In file: pyscreeze/init.py, function: def _screenshot_win32(imageFilename=None, region=None): change im = ImageGrab.grab() to im = ImageGrab.grab(all_screens= True)
• handle new introduced negative coordinates due to multiple monitor: In file: pyscreeze/init.py, function: def locateOnScreen(image, minSearchTime=0, **kwargs): behind retVal = locate(image, screenshotIm, **kwargs) add

if retVal and sys.platform == 'win32':
    # get the lowest x and y coordinate of the monitor setup
    monitors = win32api.EnumDisplayMonitors()
    x_min = min([mon[2][0] for mon in monitors])
    y_min = min([mon[2][1] for mon in monitors])
    # add negative offset due to multi monitor
    retVal = Box(left=retVal[0] + x_min, top=retVal[1] + y_min, width=retVal[2], height=retVal[3])

• don't forget to add the import win32api In file: pyscreeze/init.py,:

if sys.platform == 'win32': # TODO - Pillow now supports ImageGrab on macOS.
    import win32api # used for multi-monitor fix
    from PIL import ImageGrab

It works. Thanks alot.

[jesuspabloalfaro]

If still relevant for someone on windows:

In my opinion the issue is, that the current version of pyscreeze utilizing ImageGrab (Pillow) on windows only uses single-screen grab.

A dirty quick fix in pyscreeze could be:

• enable all_screen grabbing: In file: pyscreeze/init.py, function: def _screenshot_win32(imageFilename=None, region=None): change im = ImageGrab.grab() to im = ImageGrab.grab(all_screens= True)
• handle new introduced negative coordinates due to multiple monitor: In file: pyscreeze/init.py, function: def locateOnScreen(image, minSearchTime=0, **kwargs): behind retVal = locate(image, screenshotIm, **kwargs) add

if retVal and sys.platform == 'win32':
    # get the lowest x and y coordinate of the monitor setup
    monitors = win32api.EnumDisplayMonitors()
    x_min = min([mon[2][0] for mon in monitors])
    y_min = min([mon[2][1] for mon in monitors])
    # add negative offset due to multi monitor
    retVal = Box(left=retVal[0] + x_min, top=retVal[1] + y_min, width=retVal[2], height=retVal[3])

• don't forget to add the import win32api In file: pyscreeze/init.py,:

if sys.platform == 'win32': # TODO - Pillow now supports ImageGrab on macOS.
    import win32api # used for multi-monitor fix
    from PIL import ImageGrab

+1 on this working. Thanks pascallo

[DavidLeeberman]

@pascallo Hi Pascallo, I test Selenium for Python with PyAutoGUI on a display with 2 monitors. I modified pyscreeze/init.py as you said but then I got the following error:

File "e:\test_automation\test.py", line 90, in login_input center_x, center_y = pyautogui.locateCenterOnScreen('username_input_box.png') File "D:\Users\Jack\AppData\Local\Programs\Python\Python39\lib\site-packages\pyautogui__init.py", line 175, in wrapper return wrappedFunction(*args, **kwargs) File "D:\Users\Jack\AppData\Local\Programs\Python\Python39\lib\site-packages\pyautogui\init.py", line 207, in locateCenterOnScreen return pyscreeze.locateCenterOnScreen(*args, **kwargs) File "D:\Users\Jack\AppData\Local\Programs\Python\Python39\lib\site-packages\pyscreeze\init.py", line 408, in locateCenterOnScreen coords = locateOnScreen(image, **kwargs) File "D:\Users\Jack\AppData\Local\Programs\Python\Python39\lib\site-packages\pyscreeze\init.py", line 360, in locateOnScreen screenshotIm = screenshot(region=None) # the locateAll() function must handle cropping to return accurate coordinates, so don't pass a region here. File "D:\Users\Jack\AppData\Local\Programs\Python\Python39\lib\site-packages\pyscreeze\init__.py", line 135, in wrapper raise PyScreezeException('The Pillow package is required to use this function.') pyscreeze.PyScreezeException: The Pillow package is required to use this function.

I tried uninstall pillow and then reinstall it with the latest version 8.0.1. It's still not working. Could you please help me look into it? Thanks in advance.

[Maracaipe611]

Add these 3 lines to the top of your code to enable all monitor screengrabs in Windows:

from PIL import ImageGrab
from functools import partial
ImageGrab.grab = partial(ImageGrab.grab, all_screens=True)

[OliverTrust]

@Maracaipe611. Thanks alot. Best solution

[tgajdo]

Unfortunately this is still an issue. @Maracaipe611 suggestion doesn't seem to work for me, I still get None as response.

[manniL]

The solution from @Maracaipe611 worked fine for my dual screen setup ☺️

[hur-kyuh-leez]

@Maracaipe611 's solution works! thanks :)

[gavinlws96]

@Maracaipe611 Thank you, it worked for me! :)

[StevePic95]

    import pyautogui
    change_intense_to_calm = pyautogui.locateOnScreen('Intense.png', confidence = 0.9)
    run_script_button_x, run_script_button_y = pyautogui.center(change_intense_to_calm)
    pyautogui.click(run_script_button_x, run_script_button_y)

Hey @JohnTravolski, I know I'm a couple years late to this thread, but I wanted to share a little shortcut I recently discovered in pyautogui that you may like! The above code could also be written as follows:

    import pyautogui
    pyautogui.click('Intense.png', confidence = 0.9)

[schlechr]

Hey, I made the changes in the locateOnScreen(...) function, but this didn't work for me everywhere, as I am also using the locateAllOnScreen function. Maybe somebody needs this as well so here is what I did: I deleted the part in the locateOnScreen(...) function and instead worked in the _locateAll_opencv(...) function. At the beginning of the function I added:

if sys.platform == 'win32':
        # get the lowest x and y coordinate of the monitor setup
        monitors = win32api.EnumDisplayMonitors()
        x_min = min([mon[2][0] for mon in monitors])
        y_min = min([mon[2][1] for mon in monitors])

The reason for that is, that I already need these values when the haystackImage gets defined. I subtracted these values after the if region: so that the defined region will be used "correctly" Edit: It looks like, when you define the region like this, the return values will be sent correctly without editing them on the return at the end, therefore just reseting the min values at thsi point fixes it.

haystackImage = haystackImage[region[1]-y_min:region[1]-y_min+region[3],
                              region[0]-x_min:region[0]-x_min+region[2]]
x_min = 0
y_min = 0

And at the end of the function the values needs to be added, like it was done in the single location.

for x, y in zip(matchx, matchy):
    yield Box(x + x_min, y + y_min, needleWidth, needleHeight)

The same would be needed to be done at the _locateAll_python(...) function if you do not use OpenCV (if I understood this correctly), but for me the solution works so far.

For the screenshots to be done write, the min values must be substracted, if a region is given. Means in the _screenshot_win32(...) the following needs to be added between region = [int(x) for x in region] and im = im.crop((region[0], region[1], region[2] + region[0], region[3] + region[1])):

monitors = win32api.EnumDisplayMonitors()
x_min = min([mon[2][0] for mon in monitors])
y_min = min([mon[2][1] for mon in monitors])
region[0] -= x_min
region[1] -= y_min

P.S.: I will try to keep this comment up to date if I find anything else.

[tfau22]

Great, @pascallo. Working for me after two years. Thank you! Did you try adding this to main on pyscreeze? It's a functionallity I see everyone looking for.

[irineujba]

Looking for a solution to this problem, I visited the official documentation website and I saw this:

Q: Does PyAutoGUI work on multi-monitor setups.

A: No, right now PyAutoGUI only handles the primary monitor.

[StevePic95]

This would be amazing if it worked but I keep getting "retVal is not defined" after importing win32api and pyautogui and pyscreeze

Is this not the right way to write it?:

If I'm reading it correctly, you just need to call the locateOnScreen() function after defining it. The definition itself won't assign anything to retVal.

You may also need to take the period out of the first argument name. (I think it should just be called image if I'm reading your code right)

[nalex09]

If still relevant for someone on windows:

In my opinion the issue is, that the current version of pyscreeze utilizing ImageGrab (Pillow) on windows only uses single-screen grab.

A dirty quick fix in pyscreeze could be:

• enable all_screen grabbing: In file: pyscreeze/init.py, function: def _screenshot_win32(imageFilename=None, region=None): change im = ImageGrab.grab() to im = ImageGrab.grab(all_screens= True)
• handle new introduced negative coordinates due to multiple monitor: In file: pyscreeze/init.py, function: def locateOnScreen(image, minSearchTime=0, **kwargs): behind retVal = locate(image, screenshotIm, **kwargs) add

if retVal and sys.platform == 'win32':
    # get the lowest x and y coordinate of the monitor setup
    monitors = win32api.EnumDisplayMonitors()
    x_min = min([mon[2][0] for mon in monitors])
    y_min = min([mon[2][1] for mon in monitors])
    # add negative offset due to multi monitor
    retVal = Box(left=retVal[0] + x_min, top=retVal[1] + y_min, width=retVal[2], height=retVal[3])

• don't forget to add the import win32api In file: pyscreeze/init.py,:

if sys.platform == 'win32': # TODO - Pillow now supports ImageGrab on macOS.
    import win32api # used for multi-monitor fix
    from PIL import ImageGrab

It works. Thanks!

[pawel-j-a]

If still relevant for someone on windows:

In my opinion the issue is, that the current version of pyscreeze utilizing ImageGrab (Pillow) on windows only uses single-screen grab.

A dirty quick fix in pyscreeze could be:

• enable all_screen grabbing: In file: pyscreeze/init.py, function: def _screenshot_win32(imageFilename=None, region=None): change im = ImageGrab.grab() to im = ImageGrab.grab(all_screens= True)
• handle new introduced negative coordinates due to multiple monitor: In file: pyscreeze/init.py, function: def locateOnScreen(image, minSearchTime=0, **kwargs): behind retVal = locate(image, screenshotIm, **kwargs) add

if retVal and sys.platform == 'win32':
    # get the lowest x and y coordinate of the monitor setup
    monitors = win32api.EnumDisplayMonitors()
    x_min = min([mon[2][0] for mon in monitors])
    y_min = min([mon[2][1] for mon in monitors])
    # add negative offset due to multi monitor
    retVal = Box(left=retVal[0] + x_min, top=retVal[1] + y_min, width=retVal[2], height=retVal[3])

• don't forget to add the import win32api In file: pyscreeze/init.py,:

if sys.platform == 'win32': # TODO - Pillow now supports ImageGrab on macOS.
    import win32api # used for multi-monitor fix
    from PIL import ImageGrab

Hi, i have 3 monitors in configuration 3 : 2 : 1. Display 2 is my primary and my coords starts with 0.0 on this display. Monitor no 3 has negative coords (negative X to be clear). Your way what you type is working for me if i don't have negative coords.

If i make configuration 2 : 1 : 3 display 2 still primary and starts from 0.0 - pyautogui still working. But only if I change display no 3 to the left and make 3 : 2 : 1 with 3 as negative coords - pyautogui stops working. Will you help me? Maybe someone has the same problem as me.

[roshanboys]

If still relevant for someone on windows: I faced exactly same problem and I found another easy way around which is working for me. Since the pyautogui is always searching in primary display, x and y coordinates for it is 0,0. Easy way is to move the program where pyautogui is used to 0,0, irrespective where it is opened. I am using win32gui library for it. Here is snippet of the code which worked for me:

import win32gui import pyautogui import win32con program_window = find_window("Program Name") #Replace with program name needed.

if program_window is not None:

# Move the window to X=0, Y=0
win32gui.SetWindowPos(program_window, win32con.HWND_TOP, 0, 0, 0, 0, win32con.SWP_NOSIZE | win32con.SWP_NOZORDER)
time.sleep(1)
# Maximize the window
win32gui.ShowWindow(program_window, win32con.SW_MAXIMIZE)
time.sleep(2)
pyautogui.locateOnScreen(reference_image, grayscale=True, region=(0, 0, 1920, 1080), confidence=0.9) is not None:

def find_window(title, class_name=None): def callback(hwnd, hwnd_list): if class_name is not None: if win32gui.GetClassName(hwnd) == class_name and win32gui.IsWindowVisible(hwnd): hwnd_list.append(hwnd) else: if win32gui.GetWindowText(hwnd) == title and win32gui.IsWindowVisible(hwnd): hwnd_list.append(hwnd) return True hwnd_list = [] win32gui.EnumWindows(callback, hwnd_list) if hwnd_list: return hwnd_list[0] else: return None

By this pyautogui will always work and it is also possible to capture the original screen coordinate of the program_window and move it back after finishing the required pyautogui operations. program_rect = win32gui.GetWindowRect(program_window)

[kawsarlog]

If still relevant for someone on windows:

In my opinion the issue is, that the current version of pyscreeze utilizing ImageGrab (Pillow) on windows only uses single-screen grab.

A dirty quick fix in pyscreeze could be:

• enable all_screen grabbing: In file: pyscreeze/init.py, function: def _screenshot_win32(imageFilename=None, region=None): change im = ImageGrab.grab() to im = ImageGrab.grab(all_screens= True)
• handle new introduced negative coordinates due to multiple monitor: In file: pyscreeze/init.py, function: def locateOnScreen(image, minSearchTime=0, **kwargs): behind retVal = locate(image, screenshotIm, **kwargs) add

if retVal and sys.platform == 'win32':
    # get the lowest x and y coordinate of the monitor setup
    monitors = win32api.EnumDisplayMonitors()
    x_min = min([mon[2][0] for mon in monitors])
    y_min = min([mon[2][1] for mon in monitors])
    # add negative offset due to multi monitor
    retVal = Box(left=retVal[0] + x_min, top=retVal[1] + y_min, width=retVal[2], height=retVal[3])

• don't forget to add the import win32api In file: pyscreeze/init.py,:

if sys.platform == 'win32': # TODO - Pillow now supports ImageGrab on macOS.
    import win32api # used for multi-monitor fix
    from PIL import ImageGrab

You are a Lifesaver, this has worked perfectly for me!

[dhust]

I'm surprised there hasn't been a solution for this. Maybe the one above works for two monitors but I'm having coordinate issues with three. I use pyautogui.position() or pyautogui.displayMousePosition() and it's just not correct, which is probably also causing location issues for clicks.

[glauciofonsecaa]

If still relevant for someone on windows:

In my opinion the issue is, that the current version of pyscreeze utilizing ImageGrab (Pillow) on windows only uses single-screen grab.

A dirty quick fix in pyscreeze could be:

* enable all_screen grabbing:
  In file: pyscreeze/__init__.py, function: `def _screenshot_win32(imageFilename=None, region=None):`
  change `im = ImageGrab.grab()` to `im = ImageGrab.grab(all_screens= True)`

* handle new introduced negative coordinates due to multiple monitor:
  In file: pyscreeze/__init__.py, function: `def locateOnScreen(image, minSearchTime=0, **kwargs):` behind `retVal = locate(image, screenshotIm, **kwargs)` add

if retVal and sys.platform == 'win32':
    # get the lowest x and y coordinate of the monitor setup
    monitors = win32api.EnumDisplayMonitors()
    x_min = min([mon[2][0] for mon in monitors])
    y_min = min([mon[2][1] for mon in monitors])
    # add negative offset due to multi monitor
    retVal = Box(left=retVal[0] + x_min, top=retVal[1] + y_min, width=retVal[2], height=retVal[3])

* don't forget to add the `import win32api`
  In file: pyscreeze/__init__.py,:

if sys.platform == 'win32': # TODO - Pillow now supports ImageGrab on macOS.
    import win32api # used for multi-monitor fix
    from PIL import ImageGrab

Its work, thank you Pascallo. Python 3.12.1 pyautogui 0.9.54

[asyankeesfan]

If still relevant for someone on windows:

In my opinion the issue is, that the current version of pyscreeze utilizing ImageGrab (Pillow) on windows only uses single-screen grab.

A dirty quick fix in pyscreeze could be:

• enable all_screen grabbing: In file: pyscreeze/init.py, function: def _screenshot_win32(imageFilename=None, region=None): change im = ImageGrab.grab() to im = ImageGrab.grab(all_screens= True)
• handle new introduced negative coordinates due to multiple monitor: In file: pyscreeze/init.py, function: def locateOnScreen(image, minSearchTime=0, **kwargs): behind retVal = locate(image, screenshotIm, **kwargs) add

if retVal and sys.platform == 'win32':
    # get the lowest x and y coordinate of the monitor setup
    monitors = win32api.EnumDisplayMonitors()
    x_min = min([mon[2][0] for mon in monitors])
    y_min = min([mon[2][1] for mon in monitors])
    # add negative offset due to multi monitor
    retVal = Box(left=retVal[0] + x_min, top=retVal[1] + y_min, width=retVal[2], height=retVal[3])

• don't forget to add the import win32api In file: pyscreeze/init.py,:

if sys.platform == 'win32': # TODO - Pillow now supports ImageGrab on macOS.
    import win32api # used for multi-monitor fix
    from PIL import ImageGrab

@pascallo This (pyautogui.locateOnScreen) worked UNTIL I tried to use 'region' parameter. It seems to shift my pixels somehow. The pyautogui.position function does not seem to correlate with the region coordinates anymore. So, it works if I scan the whole screen, but not for a specific region

[asyankeesfan]

If still relevant for someone on windows: In my opinion the issue is, that the current version of pyscreeze utilizing ImageGrab (Pillow) on windows only uses single-screen grab. A dirty quick fix in pyscreeze could be:

• enable all_screen grabbing: In file: pyscreeze/init.py, function: def _screenshot_win32(imageFilename=None, region=None): change im = ImageGrab.grab() to im = ImageGrab.grab(all_screens= True)
• handle new introduced negative coordinates due to multiple monitor: In file: pyscreeze/init.py, function: def locateOnScreen(image, minSearchTime=0, **kwargs): behind retVal = locate(image, screenshotIm, **kwargs) add

if retVal and sys.platform == 'win32':
    # get the lowest x and y coordinate of the monitor setup
    monitors = win32api.EnumDisplayMonitors()
    x_min = min([mon[2][0] for mon in monitors])
    y_min = min([mon[2][1] for mon in monitors])
    # add negative offset due to multi monitor
    retVal = Box(left=retVal[0] + x_min, top=retVal[1] + y_min, width=retVal[2], height=retVal[3])

• don't forget to add the import win32api In file: pyscreeze/init.py,:

if sys.platform == 'win32': # TODO - Pillow now supports ImageGrab on macOS.
    import win32api # used for multi-monitor fix
    from PIL import ImageGrab

You are a Lifesaver, this has worked perfectly for me!

This seems to not coorespond to the "locateCenterOnScreen". Just test it and use auto.moveTo and it should put the cursor on the image of auto.locateCenterOnScreen but it doesn't. Any fix to this?