search

webfp / tor-browser-selenium

Tor Browser automation with Selenium.
MIT License
556 stars 99 forks source link

Investigate Selenium 3 compatibility #58

Closed gunesacar closed 7 years ago

gunesacar commented 8 years ago

We should check if the library needs any change to be compatible with Selenium 3. At least we'll have to add 'geckodriver' executable to the PATH for the CI tests:

We need to wait for the ESR52-based Tor Browsers to land this support.

https://travis-ci.org/gunesacar/tor-browser-selenium/jobs/149722181 Message: 'geckodriver' executable needs to be in PATH.

gunesacar commented 8 years ago

Here's a WIP branch: https://github.com/gunesacar/tor-browser-selenium/tree/tbgeckodriver

gunesacar commented 8 years ago

Import errors in Python 3.4 and 3.5 fixed with the release of Selenium 3.0.0.b3. https://travis-ci.org/gunesacar/tor-browser-selenium/builds/160031352

psivesely commented 7 years ago

:+1: for landing Selenium 3 support.

gunesacar commented 7 years ago

@fowlslegs, in case it helps with your internal scheduling, we need to wait until the release of the ESR 52 based Tor Browsers to land Selenium 3 support.

Tor Browser 6.X versions are based on ESR 45, which is not supported by geckodriver.

The expected date for ESR52 based Tor Browser is June 13: https://lists.torproject.org/pipermail/tbb-dev/2017-February/000472.html.

mjuarezm commented 7 years ago

For the record, I merged the tbgeckodriver branch onto tor-browser-crawler and tested it with the latest TBB (7.0.2). I am able to run the crawler successfully in our setting (haven't tested edge cases). However, it doesn't work for newer versions of geckodriver or/and Selenium.

gunesacar commented 7 years ago

Thanks @mjuarezm ! I guess the next step would be to check which tests fail with that branch and address them.

gunesacar commented 7 years ago

Here's the PR for the selenium 3 compatibility: https://github.com/webfp/tor-browser-selenium/pull/88

gunesacar commented 7 years ago

Fixed in https://github.com/webfp/tor-browser-selenium/pull/88#issuecomment-334484938

tadatitam commented 7 years ago

Getting this error message on torbrowser-linux64-7.0.8. Tested this with selenium 3.6, tbselenium 0.3.2 on Ubuntu 16.04 (Xenial64).

    self.driver = TorBrowserDriver(os.environ['TBB_PATH'])
  File "/home/ubunutu16/.local/lib/python2.7/site-packages/tbselenium/tbdriver.py", line 57, in __init__
    log_path=tbb_logfile_path)
  File "/home/ubunutu16/.local/lib/python2.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 144, in __init__
    self.service.start()
  File "/home/ubunutu16/.local/lib/python2.7/site-packages/selenium/webdriver/common/service.py", line 81, in start
    os.path.basename(self.path), self.start_error_message)
selenium.common.exceptions.WebDriverException: Message: 'geckodriver' executable needs to be in PATH.

The same code was working perfectly for torbrowser-linux64-7.0.6.

gunesacar commented 7 years ago

@tadatitam you need to have geckodriver executable (version 0.17.0) in your system's path starting from 0.3.2. It's required by Firefox actually.

You can download it from the following address: https://github.com/mozilla/geckodriver/releases/download/v0.17.0/geckodriver-v0.17.0-linux64.tar.gz"

You can also check the relevant MDN documents: https://developer.mozilla.org/en-US/docs/Mozilla/QA/Marionette/WebDriver#Setting_up_the_geckodriver_executable

I will open an issue to document that requirement. Sorry about that.

tadatitam commented 7 years ago

@gunesacar Ah, I see. I was getting away on Firefox by specifying executable_path='path/to/geckodriver'.

Thanks to your suggestion, I was able to get Tor to launch by adding the location of geckodriver to PATH: i.e. I can see the window showing the Tor circuit setup, and then a Firefox window opens up. However, Tor fails to open any page, and throws this new error:

    self.driver = TorBrowserDriver(os.environ['TBB_PATH'])
  File "/home/ubunutu16/.local/lib/python2.7/site-packages/tbselenium/tbdriver.py", line 57, in __init__
    log_path=tbb_logfile_path)
  File "/home/ubunutu16/.local/lib/python2.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 154, in __init__
    keep_alive=True)
  File "/home/ubunutu16/.local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 151, in __init__
    self.start_session(desired_capabilities, browser_profile)
  File "/home/ubunutu16/.local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 240, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/home/ubunutu16/.local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 308, in execute
    self.error_handler.check_response(response)
  File "/home/ubunutu16/.local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: connection refused

I'm going to try and figure out why this is happening and get back with my findings. Thanks for helping out. :)

gunesacar commented 7 years ago

This is a generic error when the geckodriver doesn't respond for one reason or other. To debug this interface you may listen to loopback interface with Wireshark. But, this is not guaranteed to give you anything.

I'd check the geckodriver version, you need to have version 0.17.0.

If you are using custom parameters for the initialization, feel free to run your code by me.

I just checked to make sure, the below code seems to work with tbselenium 0.3.2: https://gist.github.com/gunesacar/1b48b71d8619885a0dcb95bb59da592c

tadatitam commented 7 years ago

Thank you thank you. There was a mismatch in the geckodriver version. I was using 0.19.0. :)