Running tests results in errors

[alexristich]

When I run tests from the tests directory using ./run_selenium_tests.sh I'm getting errors that appear before any tests are run. Here is the printout from the latest version of the master branch:

~/privacybadgerchrome/tests ~/privacybadgerchrome/tests
scripts/makezip.sh 
scripts/makecrx.sh 
making privacy_badger-chrome-2016.9.7.crx
Created privacy_badger-chrome-2016.9.7.crx
~/privacybadgerchrome/tests
Using base prefix '/Library/Frameworks/Python.framework/Versions/3.4'
New python executable in /Users/Poseidon/privacybadgerchrome/tests/PBTESTENV/bin/python3.4
Also creating executable in /Users/Poseidon/privacybadgerchrome/tests/PBTESTENV/bin/python
Installing setuptools, pip, wheel...done.
Collecting selenium (from -r sel_requirements.txt (line 1))
  Using cached selenium-2.53.6-py2.py3-none-any.whl
Collecting xvfbwrapper (from -r sel_requirements.txt (line 2))
Collecting pytest (from -r sel_requirements.txt (line 3))
  Using cached pytest-3.0.2-py2.py3-none-any.whl
Collecting py>=1.4.29 (from pytest->-r sel_requirements.txt (line 3))
  Using cached py-1.4.31-py2.py3-none-any.whl
Installing collected packages: selenium, xvfbwrapper, py, pytest
Successfully installed py-1.4.31 pytest-3.0.2 selenium-2.53.6 xvfbwrapper-0.2.8
========================= test session starts ==========================
platform darwin -- Python 3.4.3, pytest-3.0.2, py-1.4.31, pluggy-0.3.1 -- /Users/Poseidon/privacybadgerchrome/tests/PBTESTENV/bin/python3.4
cachedir: .cache
rootdir: /Users/Poseidon/privacybadgerchrome/tests, inifile: 
collected 10 items / 2 errors 

================================ ERRORS ================================
____________ ERROR collecting selenium/localstorage_test.py ____________
PBTESTENV/lib/python3.4/site-packages/_pytest/python.py:209: in fget
    return self._obj
E   AttributeError: 'Module' object has no attribute '_obj'

During handling of the above exception, another exception occurred:
PBTESTENV/lib/python3.4/site-packages/_pytest/python.py:411: in _importtestmodule
    mod = self.fspath.pyimport(ensuresyspath=importmode)
PBTESTENV/lib/python3.4/site-packages/py/_path/local.py:650: in pyimport
    __import__(modname)
E     File "/Users/Poseidon/privacybadgerchrome/tests/selenium/localstorage_test.py", line 39
E       print self.js("return pb.storage.getBadgerStorageObject('dnt_hashes')")
E                ^
E   SyntaxError: invalid syntax
____________ ERROR collecting selenium/super_cookie_test.py ____________
PBTESTENV/lib/python3.4/site-packages/_pytest/python.py:209: in fget
    return self._obj
E   AttributeError: 'Module' object has no attribute '_obj'

During handling of the above exception, another exception occurred:
PBTESTENV/lib/python3.4/site-packages/_pytest/python.py:411: in _importtestmodule
    mod = self.fspath.pyimport(ensuresyspath=importmode)
PBTESTENV/lib/python3.4/site-packages/py/_path/local.py:650: in pyimport
    __import__(modname)
E     File "/Users/Poseidon/privacybadgerchrome/tests/selenium/super_cookie_test.py", line 32
E       print self.js("return localStorage['frameId']")
E                ^
E   SyntaxError: invalid syntax
!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!
======================= 2 error in 0.56 seconds ========================

@gunesacar @cooperq Is this happening for you guys too, or is there something I need to configure differently on my machine?

[gunesacar]

It appears that you're using Python 3 and it complains about the print statements without the parenthesis (pre-py3 syntax).

This inspired me to run the Travis tests with Python 3: Here I reproduce the issue: https://travis-ci.org/gunesacar/privacybadgerchrome/builds/161482069 Here it's fixed: https://travis-ci.org/gunesacar/privacybadgerchrome/builds/161482695

I added the commit to my existing PR so that the new code and the associated test get in at the same time. Can revert if this is not a good idea.

[alexristich]

Awesome, thanks for the quick response! I've got a branch that's just about ready for a PR, though I want to make sure it's working with the latest commits from the master branch. I'll wait until your latest PR gets merged in before I submit my own PR, that way I can make sure I didn't break anything and also potentially update any tests that need to be expanded :)

[cooperq]

I have merged @gunesacar's branch. You should be good to go!

[alexristich]

Hmm, looks like test_removed_origin_display is failing on the current master branch. Should I create a new issue for this, or is it possibly related to the changes in the tests?

[gunesacar]

Should I create a new issue for this, or is it possibly related to the changes in the tests?

It's hard to say with the available info. Linking to the failed Travis tests would help a lot. We didn't touch that test at all, so I'd be surprised if it's related to last changes.

The master branch doesn't fail on Travis: https://travis-ci.org/EFForg/privacybadgerchrome/builds. So do my local tests...

Feel free to file an issue if the failure is persistent. I'll be happy to help out.

[alexristich]

Very interesting; it's definitely a persistent issue when I run it locally. I'm not too familiar with the current test suite (and haven't personally meddled with Travis before), so if it's something you can help pinpoint more quickly that'd be awesome.

In the printout, I notice specifically that it mentions this: response = {'sessionId': 'e710e2dff024d10abd5507c6f70afdf6', 'status': 13, 'value': {'message': 'unknown error: cannot determine ... (Driver info: chromedriver=2.21.371459 (36d3d07f660ff2bc1bf28a75d1cdabed0983e7c4),platform=Mac OS X 10.11.6 x86_64)'}}

I'm wondering if this is any indication of where the issue might lie. Here is the full printout from the console. The test that is failing specifically is the one initialized on line 352 here: https://travis-ci.org/EFForg/privacybadgerchrome/jobs/161513080

=========================== FAILURES ===========================
_________ OptionsPageTest.test_removed_origin_display __________

self = <options_test.OptionsPageTest testMethod=test_removed_origin_display>

    def test_removed_origin_display(self):
        """Ensure origin is displayed and removed properly."""
        self.add_test_origin("pbtest.org", "block")

        self.load_url(pbtest.PB_CHROME_OPTIONS_PAGE_URL)
        origins = self.driver.find_element_by_id("blockedResourcesInner")

        # Remove displayed origin.
        try:
            remove_origin_element = origins.find_element_by_xpath(
                './/div[@data-origin="pbtest.org"]' +
                '//div[@class="removeOrigin"]')
        except NoSuchElementException:
            self.fail("Tracking origin is not displayed")
>       remove_origin_element.click()

selenium/options_test.py:122: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium/webdriver/remote/webelement.py:72: in click
    self._execute(Command.CLICK_ELEMENT)
/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium/webdriver/remote/webelement.py:461: in _execute
    return self._parent.execute(command, params)
/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium/webdriver/remote/webdriver.py:236: in execute
    self.error_handler.check_response(response)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <selenium.webdriver.remote.errorhandler.ErrorHandler object at 0x103edee48>
response = {'sessionId': 'e710e2dff024d10abd5507c6f70afdf6', 'status': 13, 'value': {'message': 'unknown error: cannot determine ... (Driver info: chromedriver=2.21.371459 (36d3d07f660ff2bc1bf28a75d1cdabed0983e7c4),platform=Mac OS X 10.11.6 x86_64)'}}

    def check_response(self, response):
        """
            Checks that a JSON response from the WebDriver does not have an error.

            :Args:
             - response - The JSON response from the WebDriver server as a dictionary
               object.

            :Raises: If the response contains an error message.
            """
        status = response.get('status', None)
        if status is None or status == ErrorCode.SUCCESS:
            return

        value = None
        message = response.get("message", "")
        screen = response.get("screen", "")
        stacktrace = None
        if isinstance(status, int):
            value_json = response.get('value', None)
            if value_json and isinstance(value_json, basestring):
                import json
                try:
                    value = json.loads(value_json)
                    status = value.get('error', None)
                    if status is None:
                        status = value["status"]
                        message = value["value"]
                        if not isinstance(message, basestring):
                            value = message
                            try:
                                message = message['message']
                            except TypeError:
                                message = None
                    else:
                        message = value.get('message', None)
                except ValueError:
                    pass

        exception_class = ErrorInResponseException
        if status in ErrorCode.NO_SUCH_ELEMENT:
            exception_class = NoSuchElementException
        elif status in ErrorCode.NO_SUCH_FRAME:
            exception_class = NoSuchFrameException
        elif status in ErrorCode.NO_SUCH_WINDOW:
            exception_class = NoSuchWindowException
        elif status in ErrorCode.STALE_ELEMENT_REFERENCE:
            exception_class = StaleElementReferenceException
        elif status in ErrorCode.ELEMENT_NOT_VISIBLE:
            exception_class = ElementNotVisibleException
        elif status in ErrorCode.INVALID_ELEMENT_STATE:
            exception_class = InvalidElementStateException
        elif status in ErrorCode.INVALID_SELECTOR \
                or status in ErrorCode.INVALID_XPATH_SELECTOR \
                or status in ErrorCode.INVALID_XPATH_SELECTOR_RETURN_TYPER:
            exception_class = InvalidSelectorException
        elif status in ErrorCode.ELEMENT_IS_NOT_SELECTABLE:
            exception_class = ElementNotSelectableException
        elif status in ErrorCode.INVALID_COOKIE_DOMAIN:
            exception_class = WebDriverException
        elif status in ErrorCode.UNABLE_TO_SET_COOKIE:
            exception_class = WebDriverException
        elif status in ErrorCode.TIMEOUT:
            exception_class = TimeoutException
        elif status in ErrorCode.SCRIPT_TIMEOUT:
            exception_class = TimeoutException
        elif status in ErrorCode.UNKNOWN_ERROR:
            exception_class = WebDriverException
        elif status in ErrorCode.UNEXPECTED_ALERT_OPEN:
            exception_class = UnexpectedAlertPresentException
        elif status in ErrorCode.NO_ALERT_OPEN:
            exception_class = NoAlertPresentException
        elif status in ErrorCode.IME_NOT_AVAILABLE:
            exception_class = ImeNotAvailableException
        elif status in ErrorCode.IME_ENGINE_ACTIVATION_FAILED:
            exception_class = ImeActivationFailedException
        elif status in ErrorCode.MOVE_TARGET_OUT_OF_BOUNDS:
            exception_class = MoveTargetOutOfBoundsException
        else:
            exception_class = WebDriverException
        if value == '' or value is None:
            value = response['value']
        if isinstance(value, basestring):
            if exception_class == ErrorInResponseException:
                raise exception_class(response, value)
            raise exception_class(value)
        if message == "" and 'message' in value:
            message = value['message']

        screen = None
        if 'screen' in value:
            screen = value['screen']

        stacktrace = None
        if 'stackTrace' in value and value['stackTrace']:
            stacktrace = []
            try:
                for frame in value['stackTrace']:
                    line = self._value_or_default(frame, 'lineNumber', '')
                    file = self._value_or_default(frame, 'fileName', '<anonymous>')
                    if line:
                        file = "%s:%s" % (file, line)
                    meth = self._value_or_default(frame, 'methodName', '<anonymous>')
                    if 'className' in frame:
                        meth = "%s.%s" % (frame['className'], meth)
                    msg = "    at %s (%s)"
                    msg = msg % (meth, file)
                    stacktrace.append(msg)
            except TypeError:
                pass
        if exception_class == ErrorInResponseException:
            raise exception_class(response, message)
        elif exception_class == UnexpectedAlertPresentException and 'alert' in value:
            raise exception_class(message, screen, stacktrace, value['alert'].get('text'))
>       raise exception_class(message, screen, stacktrace)
E       selenium.common.exceptions.WebDriverException: Message: unknown error: cannot determine loading status
E       from unexpected alert open
E         (Session info: chrome=53.0.2785.116)
E         (Driver info: chromedriver=2.21.371459 (36d3d07f660ff2bc1bf28a75d1cdabed0983e7c4),platform=Mac OS X 10.11.6 x86_64)

/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium/webdriver/remote/errorhandler.py:192: WebDriverException

[alexristich]

@gunesacar - I should note too that I don't know how to rerun a Travis test on a build. You can see my branch user_allow here: https://travis-ci.org/EFForg/privacybadgerchrome/branches. The failed test here was due to a timeout it appears; is there a way for me to get this test to run again without pushing a new commit, or should I just push a new commit with a minor change to get it to run again?

Thanks again for helping with this. I'm sure I'll get it figured out soon!

[gunesacar]

Hi @alexristich,

This may be https://bugs.chromium.org/p/chromedriver/issues/detail?id=1362 Can you tell me the version of your chromedriver? (chromedriver --version) And can you please update and retest if it's not the most recent.

To run the Travis tests without pushing to upstream, you should fork the repo and push to your own remote. You should also sign in to Travis and allow it to run the tests for your repo. It's extremely easy to use once you set it up.

[cooperq]

@alexristich do you have some alert code in there? I think you should look at this portion of your error output: selenium.common.exceptions.WebDriverException: Message: unknown error: cannot determine loading status from unexpected alert open

[alexristich]

@gunesacar You're a genius. :) Updated chromedriver from 2.21.x to 2.24.x and got this:

==================== 15 passed in 134.92 seconds ====================

(Thanks for your help too, @cooperq! Glad I didn't have to debug further for now... :P)