search

asweigart / pyautogui

A cross-platform GUI automation Python module for human beings. Used to programmatically control the mouse & keyboard.
BSD 3-Clause "New" or "Revised" License
10.24k stars 1.24k forks source link

pyautogui not reading screen pixels #280

Open executionByFork opened 5 years ago

executionByFork commented 5 years ago

I am having an odd issue where pyautogui isn't properly reading the pixels on my screen. I believe this is what is causing the locateOnScreen() function to not work at all for me. Basically, every call to pyautogui.pixel(x, y) returns (0,0,0) for any x and y.

>>> import pyautogui
>>> pyautogui.position()
(965, 442)
>>> pyautogui.pixel(965,442)
(0, 0, 0)
>>> pyautogui.locateOnScreen('button.png')
>>>

In the above, I got the location of my mouse which was hovering over a blue icon, and tried to read the RGB values of the pixel. I've tried several different colors and locations on my screen, all return the same (0,0,0) output. I have tried using the region parameter, grayscale, and even confidence (which uses opencv-python) to get locateOnScreen() to work without any luck.

Originally, I was curious if it there was something wrong with the hardware of my laptop. I am running Ubuntu on a machine that was originally a Windows install. So I tried the following with PIL

>>> from PIL import Image
>>> im = Image.open('../screen.png')
>>> pix = im.load()
>>> print(im.size)
(1366, 768)
>>> print(pix[1048,118])
(35, 39, 41, 255)

The PIL library works as it's supposed to, so it's not some sort of odd hardware issue. There is something wrong with the pyautogui library, which is quite frustrating because I need the library's image recognition to work for my script. Any suggestions on how to fix this?

I am running Ubuntu 18.04 LTS, and it is essentially a fresh install.

executionByFork commented 5 years ago 
$ pip freeze
numpy==1.15.4
opencv-python==3.4.3.18
Pillow==5.3.0
PyAutoGUI==0.9.38
PyMsgBox==1.0.6
PyScreeze==0.1.18
PyTweening==1.0.3
six==1.11.0
xlib==0.21

Additionally, please see my other issue I opened. They might be related #279

Ohmano1 commented 5 years ago

I have the same problems with locateonscreen which os u use?

executionByFork commented 5 years ago

Also, I have just found that pyautogui.screenshot() returns a black image. This may be the core of the problem.

>>> im = pyautogui.screenshot()
>>> pix = im.load()
>>> for x in range(1300):
...     for y in range(700):
...             if pix[x,y] != (0,0,0):
...                     print(pix[x,y])
... 
>>>
executionByFork commented 5 years ago

Still having this problem... I tried using pyautogui on my other laptop running PureOS. Nothing is different. The screenshots this module takes are just black screens.

$ python3
Python 3.6.7 (default, Oct 21 2018, 08:08:16) 
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyautogui
>>> im1 = pyautogui.screenshot()
>>> im1.save('test.png')
>>>

The screenshot: test

$ pip freeze
Pillow==5.3.0
PyAutoGUI==0.9.38
PyMsgBox==1.0.6
PyScreeze==0.1.18
PyTweening==1.0.3
six==1.11.0
xlib==0.21

I'm not understanding how I am having the exact same issue on two separate laptops running separate operating systems, while no one else has reported issues like this. What am I doing wrong? I installed the module via pip inside of a virtual environment. I haven't seen anything in the documentation that says I need to do anything else

valizas commented 5 years ago

I think the problem is that pyautogui doesn't come to terms with wayland yet, the x11 substitute.

dmenig commented 3 years ago

I have the same issue. Have you been able to solve the balck screen problem ?

henry-md commented 3 years ago

I have a very similar problem - pyautogui.pixel gives me garbage values, for ex. (43, 45, 23) when the pixel is clearly white, or (34, 200, 10) when it's more blue than anything. I've tried the script on other people's macs, and I get the same garbage values (and no errors). Anyone know if this could possibly be a problem with pyautogui?

chauanphu commented 3 years ago

It seems that after you install the pyautogui and its following packages, you should restart the computer. At first my pyautogui on my ubuntu doesn't read pixel nor interact with mouse. But after restarting, now it works like a charm

NivEz commented 3 years ago

I have a very similar problem - pyautogui.pixel gives me garbage values, for ex. (43, 45, 23) when the pixel is clearly white, or (34, 200, 10) when it's more blue than anything. I've tried the script on other people's macs, and I get the same garbage values (and no errors). Anyone know if this could possibly be a problem with pyautogui?

I had the same problem, I think that the restart chauanphu talked about solved the problem.

chauanphu commented 3 years ago

When I tried to run the function pyautogui.displayMousePosition() as root, it reported this error:

error.DisplayConnectionError(self.display_name, r.reason)
Xlib.error.DisplayConnectionError: Can't connect to display ":0": b'Authorization required, but no authorization protocol specified

Hope this would be helpful !

henry-md commented 3 years ago

When I tried to run the function pyautogui.displayMousePosition() as root, it reported this error:

error.DisplayConnectionError(self.display_name, r.reason)
Xlib.error.DisplayConnectionError: Can't connect to display ":0": b'Authorization required, but no authorization protocol specified
  1. sudo nano /etc/gdm3/custom.conf
  2. Remove # in #WaylandEnable=false to force your computer to use x11
  3. Restart the computer

Hope this would be helpful !

There's no pyautogui function displayMousePosition() that I know of - only .position() (Reference: https://pyautogui.readthedocs.io/en/latest/mouse.html)

That's even besides the point. Pyautogui is so slow at basically everything so if you can use another library for any tasks, do it. I would do this instead:

from pynput.mouse import Button, Controller mouse = Controller() (x, y) = mouse.position

Im-Josepaul commented 3 years ago

When I tried to run the function pyautogui.displayMousePosition() as root, it reported this error:

error.DisplayConnectionError(self.display_name, r.reason)
Xlib.error.DisplayConnectionError: Can't connect to display ":0": b'Authorization required, but no authorization protocol specified
  1. sudo nano /etc/gdm3/custom.conf
  2. Remove # in #WaylandEnable=false to force your computer to use x11
  3. Restart the computer

Hope this would be helpful !

Mahn....This is the solution for me! When I did this, while restarting, my laptop froze, but I managed to get back into linux and is working fine.

jrams312 commented 2 years ago

When I tried to run the function pyautogui.displayMousePosition() as root, it reported this error:

error.DisplayConnectionError(self.display_name, r.reason)
Xlib.error.DisplayConnectionError: Can't connect to display ":0": b'Authorization required, but no authorization protocol specified
* A website explains that the trouble is cause because pyautogui has problem with wayland. So after searching on website, I found this useful link [https://askubuntu.com/questions/961304/how-do-you-switch-from-wayland-back-to-xorg-in-ubuntu-17-10](url)

* Just switch the display protocol from wayland back to x11:

1. `sudo nano /etc/gdm3/custom.conf`

2. Remove # in `#WaylandEnable=false` to force your computer to use x11

3. Restart the computer

Hope this would be helpful !

That did the trick! Thanks.

Yona-TYT commented 1 month ago

https://www.phoronix.com/news/Wayland-Protocols-1.37

"The other two new protocols are ext-image-capture-source and ext-image-copy-capture that allow for capturing the pixel content of outputs and top-level windows. "