webfp / tor-browser-selenium

Tor Browser automation with Selenium.
MIT License
528 stars 101 forks source link

Compatible versions - WebDriverException Message: Process unexpectedly closed with status 255 #189

Closed GraemeMeyerGT closed 1 year ago

GraemeMeyerGT commented 1 year ago

Has something changed in the last few weeks? My code was working with tbselenium 0.6.3, and tor-browser 12.0.5, but now no longer and I get the classic status 255 error.

I've tried upgrading to tor-browser 12.0.7, tried every major version of geckodriver 0.30 - 0.33 and I cannot find a compatible combination.

Can you list a compatible version combination of tbselenium, tor browser, selenium and geckodriver?

I see you changed the CI url for geckodriver recently which is what's making me wonder if something has changed, other than the most recent tor-browser version.

The error:

Exception has occurred: WebDriverException
Message: Process unexpectedly closed with status 255
...
    main()
selenium.common.exceptions.WebDriverException: Message: Process unexpectedly closed with status 255

I have reset and tried with very simple code sample from your examples, but same error:

from tbselenium.tbdriver import TorBrowserDriver
with TorBrowserDriver("/opt/tor-browser/") as driver:
    driver.get('https://check.torproject.org')

I'm using Ubuntu 22.04 LTS headless. Tried also a slightly more complex example using pyvirtualdisplay (as we were doing in our production code):

from tbselenium.tbdriver import TorBrowserDriver
from tbselenium.utils import start_xvfb, stop_xvfb
from os.path import join, dirname, realpath

def headless_visit(tbb_dir):
    out_img = join(dirname(realpath(__file__)), "headless_screenshot.png")
    # start a virtual display
    xvfb_display = start_xvfb()
    with TorBrowserDriver(tbb_dir) as driver:
        driver.load_url("https://check.torproject.org")
        driver.get_screenshot_as_file(out_img)
        print("Screenshot is saved as %s" % out_img)

    stop_xvfb(xvfb_display)

def main():
    headless_visit("/opt/tor-browser")

if __name__ == '__main__':
    main()

But same result. I really can't figure out what's changed. If you could help in any way at all, it would be much appreciated.

Edit: When trying new versions of geckodriver, I've been copying them to /usr/local/bin/geckodriver and ensuring file ownership and execute permission. That dir is on the path in Ubuntu 22, so that should I think be enough to set the geckodriver versions, but if there's a switch in tbselenium to do it manually, that would be worth testing as well just in case. I can't see such an option however.

gunesacar commented 1 year ago

The following combination works on Travis CI and on my local 23.04 Ubuntu. Please make sure you upgrade your selenium Python package as well:

In similar future cases, you may check the CI logs to see what version of packages are used in the tests: https://api.travis-ci.com/v3/job/603956148/log.txt

GraemeMeyerGT commented 1 year ago

Ah I was behind in Selenium version (4.9.1). However, once upgraded to 4.10.0, I now receive following error:

/bin/python /opt/test/test.py
Traceback (most recent call last):
  File "/opt/test/test.py", line 21, in <module>
    main()
  File "/opt/test/test.py", line 17, in main
    headless_visit("/opt/tor-browser")
  File "/opt/test/test.py", line 9, in headless_visit
    with TorBrowserDriver(tbb_dir) as driver:
  File "/home/ctiuser/.local/lib/python3.10/site-packages/tbselenium/tbdriver.py", line 98, in __init__
    super(TorBrowserDriver, self).__init__(
TypeError: WebDriver.__init__() got an unexpected keyword argument 'executable_path'

Versions:

geckodriver --version
geckodriver 0.31.0 (b617178ef491 2022-04-06 11:57 +0000)

python --version
Python 3.10.6

pip list 
Package                Version
---------------------- -------------
...
PyVirtualDisplay       3.0
...
selenium               4.10.0
...
tbselenium             0.6.3
...

That's for the following code sample:

from tbselenium.tbdriver import TorBrowserDriver
from tbselenium.utils import start_xvfb, stop_xvfb
from os.path import join, dirname, realpath

def headless_visit(tbb_dir):
    out_img = join(dirname(realpath(__file__)), "headless_screenshot.png")
    # start a virtual display
    xvfb_display = start_xvfb()
    with TorBrowserDriver(tbb_dir) as driver:
        driver.load_url("https://check.torproject.org")
        driver.get_screenshot_as_file(out_img)
        print("Screenshot is saved as %s" % out_img)

    stop_xvfb(xvfb_display)

def main():
    headless_visit("/opt/tor-browser")

if __name__ == '__main__':
    main()

Has the order/default arguments for TorBrowserDriver changed? I had thought that tbb directory path supplied as the only argument was previously valid

GraemeMeyerGT commented 1 year ago

Ah I see this is likely fixed in https://github.com/webfp/tor-browser-selenium/commit/0254df106f11d7d8269d2e2c6715ad55073a1aef. (and same as #187) - is a version containing that fix due for release any time soon? Should I try to use that commit directly?

gunesacar commented 1 year ago

Will do a release by the end of day today. Thanks for pointing it out.

GraemeMeyerGT commented 1 year ago

Ah wonderful. Thank you! 🙏

gunesacar commented 1 year ago

New release is up: https://pypi.org/project/tbselenium/0.7.0/

Thanks for filing the issue.

gunesacar commented 1 year ago

Feel free to reopen if you still see issues.

GraemeMeyerGT commented 1 year ago

Thanks @gunesacar. I am still experiencing the issues, but I think I have narrowed it down to display issues. I have a full desktop Ubuntu 22 with GUI where it works fine, but I cannot get it working on the headless Ubuntu 22 used in Azure. Even using xvfb_display = start_xvfb() as in the headless example doesn't work.

Currently trying various iterations of virtual displays, xfce4 etc. to do some debugging, but it's slow going. I did have success by installing a full desktop environment and rdp server (per Azure instructions), and I think I also had to install the full Firefox package.

I'm sure this isn't really your problem or anything to do with your software, but if you have any tips for making it work, they would be really appreciated. You wouldn't think it would be difficult to run a headless program on headless server 😕

GraemeMeyerGT commented 1 year ago

Okay I've given up trying to keep this headless, and am just installing a desktop environment on Ubuntu to keep the Tor Browser happy.

From the Azure instructions, all that is required is:

sudo apt-get update
sudo DEBIAN_FRONTEND=noninteractive apt-get -y install xfce4
sudo apt install xfce4-session

I was previously working with a light version of xfce4, which introduced further problems, so despite the fact the full version is quite heavy, if anyone in future wants to avoid the trouble, I suggest installing the full xfce4 version.

With that installed, all seems to work well. Thanks once again.