escattone
commented
6 years ago Closed limed closed 6 years ago
escattone
commented
6 years ago escattone
commented
6 years ago Yesterday, I created and attached 4 new AWS ACM certificates, one for each of the following:
I removed devmo.developer.mozilla.org from the certs for the primary prod CDN and the web ELB, since it has not been in our allowed hosts for at least the past year and doesn't resolve anyway.
escattone
commented
6 years ago The new prod PoC instance is running in maintenance mode and available for testing. It is using a dump of the prod DB from two days ago.
escattone
commented
6 years ago Load https://developer-prod.mdn.mozit.cloud/en-US/
Load https://developer-prod.mdn.mozit.cloud/en-US/docs/Web/HTML
escattone
commented
6 years ago With a Kuma environment and Docker, run the functional tests:
scripts/run_functional_tests.sh --maintenance-mode --base-url https://developer-prod.mdn.mozit.cloud -m "not login" tests/functionalThis will run a selection of safe Selenium tests (no login, no page edits) with Chrome and Firefox, running in Docker images. A failing test is automatically re-run before calling it a failure. Test results are created, with screenshots, tracebacks, and logs for failing tests, including xfail and rerun tests.
Details:
(mdntest) rjohnson-25186:kuma rjohnson$ scripts/run_functional_tests.sh --maintenance-mode --base-url https://developer-prod.mdn.mozit.cloud -m "not login" tests/functional
*** Building integration tests image...
Sending build context to Docker daemon 179.3MB
Step 1/10 : FROM python:2.7-slim
2.7-slim: Pulling from library/python
Digest: sha256:0a43a6d7858af4a42427c792b682936d2cd34e183fb026627f53ddb556d4bf62
Status: Image is up to date for python:2.7-slim
---> c9cde4658340
Step 2/10 : WORKDIR /app
---> Using cache
---> fda1425b260f
Step 3/10 : RUN set -ex && apt-get update && apt-get install -y --no-install-recommends mime-support build-essential libxml2-dev libxslt1.1 libxslt1-dev zlib1g-dev && rm -rf /var/lib/apt/lists/*
---> Using cache
---> c7cd96772567
Step 4/10 : ENV PYTEST_PROCESSES 5
---> Using cache
---> 6a8ffee2f698
Step 5/10 : ENV PRIVACY "public restricted"
---> Using cache
---> 32ac9e9d4741
Step 6/10 : ENV TESTS_PATH /app/tests
---> Using cache
---> 43a2778303ec
Step 7/10 : ENV RESULTS_PATH /app/results
---> Using cache
---> 0c3166366abb
Step 8/10 : COPY ./requirements /app/requirements
---> Using cache
---> 4d3ee7eb9284
Step 9/10 : RUN pip install --no-cache-dir -r requirements/test.txt
---> Using cache
---> 29fc609364e0
Step 10/10 : COPY tests /app/tests
---> 3976ba7b1070
Successfully built 3976ba7b1070
Successfully tagged kuma-integration-tests:latest
*** Running dockerized chrome...
dd589bea217129e241bd100ca176e699f38feabe3fc924c7ef2edfa36de22977
*** Running dockerized firefox...
30eceeb4e436298924c64cfe882a901387a3f849fd3b7fdefe0743ea071e358e
*** Running integration tests against chrome...
=============================================== test session starts ===============================================
platform linux2 -- Python 2.7.15, pytest-3.1.3, py-1.4.33, pluggy-0.4.0
driver: Remote
sensitiveurl: .* *** WARNING: sensitive url matches https://developer-prod.mdn.mozit.cloud ***
metadata: {'Python': '2.7.15', 'Driver': 'Remote', 'Capabilities': {'browserName': 'chrome'}, 'Server': 'hub:4444', 'Base URL': 'https://developer-prod.mdn.mozit.cloud', 'Platform': 'Linux-4.9.93-linuxkit-aufs-x86_64-with-debian-9.5', 'kuma': {u'services': {u'search': {u'available': True, u'count': 60165, u'populated': True}, u'kumascript': {u'available': True, u'revision': u'e57e27c371416c7e104f7843318e9166dbe3c1f3'}, u'test_accounts': {u'available': False}, u'database': {u'available': True, u'document_count': 130018, u'populated': True}}, u'version': 1, u'request': {u'url': u'https://developer-prod.mdn.mozit.cloud/_kuma_status.json', u'is_secure': True, u'host': u'developer-prod.mdn.mozit.cloud', u'scheme': u'https'}, 'response': {'headers': {'X-Cache': 'Miss from cloudfront', 'strict-transport-security': 'max-age=63072000', 'x-content-type-options': 'nosniff', 'Content-Language': 'en-US', 'Content-Encoding': 'gzip', 'Transfer-Encoding': 'chunked', 'Expires': 'Wed, 12 Sep 2018 22:01:35 GMT', 'Vary': 'Accept-Encoding', 'Server': 'meinheld/0.6.1', 'Connection': 'keep-alive', 'x-xss-protection': '1; mode=block', 'X-Amz-Cf-Id': 'n5WOJjNVEf5pXffVAELwdVxd2l52uo3MPjd1dioWUkwXgCXFM7MYOg==', 'Cache-Control': 'no-cache, no-store, must-revalidate, max-age=0', 'Date': 'Wed, 12 Sep 2018 22:01:35 GMT', 'X-Frame-Options': 'DENY', 'Content-Type': 'application/json', 'Via': '1.1 27fdab293d6dff03ec20e408968c606d.cloudfront.net (CloudFront)'}}, u'settings': {u'ATTACHMENT_HOST': u'demos.mdn.mozit.cloud', u'PROTOCOL': u'https://', u'INTERACTIVE_EXAMPLES_BASE': u'https://interactive-examples.mdn.mozit.cloud', u'MAINTENANCE_MODE': True, u'STATIC_URL': u'https://developer-prod.mdn.mozit.cloud/static/', u'SITE_URL': u'https://developer-prod.mdn.mozit.cloud', u'ATTACHMENT_ORIGIN': u'prod.mdn.mozit.cloud', u'DEBUG': False, u'ALLOWED_HOSTS': [u'developer.mozilla.org', u'cdn.mdn.mozilla.net', u'mdn.mozillademos.org', u'demos.mdn.mozit.cloud', u'developer-prod.mdn.mozit.cloud', u'prod.mdn.mozit.cloud', u'developer.mozilla.com', u'mdn.mozilla.org', u'developer-new.mozilla.org', u'developers.mozilla.org'], u'REVISION_HASH': u'0591dd661bfa5e34f19a74428dff5cc8a172998c', u'LEGACY_HOSTS': [u'cdn.mdn.mozilla.net', u'developer.mozilla.com', u'mdn.mozilla.org', u'developer-new.mozilla.org', u'developers.mozilla.org']}}, 'Plugins': {'variables': '1.7.1', 'selenium': '1.11.4', 'rerunfailures': '2.1.0', 'html': '1.16.1', 'base-url': '1.4.1', 'metadata': '1.5.1'}, 'Packages': {'py': '1.4.33', 'pytest': '3.1.3', 'pluggy': '0.4.0'}}
baseurl: https://developer-prod.mdn.mozit.cloud
rootdir: /app, inifile:
plugins: selenium-1.11.4, html-1.16.1, rerunfailures-2.1.0, variables-1.7.1, metadata-1.5.1, base-url-1.4.1
collected 119 items
tests/functional/test_article.py ..s..s...
tests/functional/test_article_edit.py s
tests/functional/test_article_revision.py ..
tests/functional/test_content_experiment.py ss
tests/functional/test_dashboard.py ...x......
tests/functional/test_feedback.py ...
tests/functional/test_home.py .....s..
tests/functional/test_language_selector.py .
tests/functional/test_maintenance_mode_redirects.py ...............................................................
tests/functional/test_notfound.py ...
tests/functional/test_profiles.py .
tests/functional/test_report.py ..
tests/functional/test_search.py .......
------------------------------------ generated html file: /results/pytest.html ------------------------------------
=============================================== 7 tests deselected ================================================
======================== 105 passed, 6 skipped, 7 deselected, 1 xfailed in 198.31 seconds =========================
*** Running integration tests against firefox...
=============================================== test session starts ===============================================
platform linux2 -- Python 2.7.15, pytest-3.1.3, py-1.4.33, pluggy-0.4.0
driver: Remote
sensitiveurl: .* *** WARNING: sensitive url matches https://developer-prod.mdn.mozit.cloud ***
metadata: {'Python': '2.7.15', 'Driver': 'Remote', 'Capabilities': {'browserName': 'firefox'}, 'Server': 'hub:4444', 'Base URL': 'https://developer-prod.mdn.mozit.cloud', 'Platform': 'Linux-4.9.93-linuxkit-aufs-x86_64-with-debian-9.5', 'kuma': {u'services': {u'search': {u'available': True, u'count': 60165, u'populated': True}, u'kumascript': {u'available': True, u'revision': u'e57e27c371416c7e104f7843318e9166dbe3c1f3'}, u'test_accounts': {u'available': False}, u'database': {u'available': True, u'document_count': 130018, u'populated': True}}, u'version': 1, u'request': {u'url': u'https://developer-prod.mdn.mozit.cloud/_kuma_status.json', u'is_secure': True, u'host': u'developer-prod.mdn.mozit.cloud', u'scheme': u'https'}, 'response': {'headers': {'X-Cache': 'Miss from cloudfront', 'strict-transport-security': 'max-age=63072000', 'x-content-type-options': 'nosniff', 'Content-Language': 'en-US', 'Content-Encoding': 'gzip', 'Transfer-Encoding': 'chunked', 'Expires': 'Wed, 12 Sep 2018 22:04:55 GMT', 'Vary': 'Accept-Encoding', 'Server': 'meinheld/0.6.1', 'Connection': 'keep-alive', 'x-xss-protection': '1; mode=block', 'X-Amz-Cf-Id': 'T2PwuuXOTg7VCUXiEJMp0AxsK_q1GdQXwijkdCmoU4IYtMlhKYo6pg==', 'Cache-Control': 'no-cache, no-store, must-revalidate, max-age=0', 'Date': 'Wed, 12 Sep 2018 22:04:55 GMT', 'X-Frame-Options': 'DENY', 'Content-Type': 'application/json', 'Via': '1.1 afb9be97319013ab1a18f338fce40f2a.cloudfront.net (CloudFront)'}}, u'settings': {u'ATTACHMENT_HOST': u'demos.mdn.mozit.cloud', u'PROTOCOL': u'https://', u'INTERACTIVE_EXAMPLES_BASE': u'https://interactive-examples.mdn.mozit.cloud', u'MAINTENANCE_MODE': True, u'STATIC_URL': u'https://developer-prod.mdn.mozit.cloud/static/', u'SITE_URL': u'https://developer-prod.mdn.mozit.cloud', u'ATTACHMENT_ORIGIN': u'prod.mdn.mozit.cloud', u'DEBUG': False, u'ALLOWED_HOSTS': [u'developer.mozilla.org', u'cdn.mdn.mozilla.net', u'mdn.mozillademos.org', u'demos.mdn.mozit.cloud', u'developer-prod.mdn.mozit.cloud', u'prod.mdn.mozit.cloud', u'developer.mozilla.com', u'mdn.mozilla.org', u'developer-new.mozilla.org', u'developers.mozilla.org'], u'REVISION_HASH': u'0591dd661bfa5e34f19a74428dff5cc8a172998c', u'LEGACY_HOSTS': [u'cdn.mdn.mozilla.net', u'developer.mozilla.com', u'mdn.mozilla.org', u'developer-new.mozilla.org', u'developers.mozilla.org']}}, 'Plugins': {'variables': '1.7.1', 'selenium': '1.11.4', 'rerunfailures': '2.1.0', 'html': '1.16.1', 'base-url': '1.4.1', 'metadata': '1.5.1'}, 'Packages': {'py': '1.4.33', 'pytest': '3.1.3', 'pluggy': '0.4.0'}}
baseurl: https://developer-prod.mdn.mozit.cloud
rootdir: /app, inifile:
plugins: selenium-1.11.4, html-1.16.1, rerunfailures-2.1.0, variables-1.7.1, metadata-1.5.1, base-url-1.4.1
collected 119 items
tests/functional/test_article.py ..s..s...
tests/functional/test_article_edit.py s
tests/functional/test_article_revision.py ..
tests/functional/test_content_experiment.py ss
tests/functional/test_dashboard.py ..FEX......
tests/functional/test_feedback.py ...
tests/functional/test_home.py .....s..
tests/functional/test_language_selector.py .
tests/functional/test_maintenance_mode_redirects.py ...............................................................
tests/functional/test_notfound.py ...
tests/functional/test_profiles.py .
tests/functional/test_report.py ..
tests/functional/test_search.py F......
------------------------------------ generated html file: /results/pytest.html ------------------------------------
===================================================== ERRORS ======================================================
________________________________ ERROR at teardown of test_dashboard_load_page_two ________________________________
request = <SubRequest 'driver' for <Function 'test_dashboard_load_page_two'>>
driver_class = <class 'selenium.webdriver.remote.webdriver.WebDriver'>
driver_kwargs = {'browser_profile': None, 'command_executor': 'http://hub:4444/wd/hub', 'desired_capabilities': {'browserName': 'firefox'}}
@pytest.yield_fixture
def driver(request, driver_class, driver_kwargs):
"""Returns a WebDriver instance based on options and capabilities"""
driver = driver_class(**driver_kwargs)
event_listener = request.config.getoption('event_listener')
if event_listener is not None:
# Import the specified event listener and wrap the driver instance
mod_name, class_name = event_listener.rsplit('.', 1)
mod = __import__(mod_name, fromlist=[class_name])
event_listener = getattr(mod, class_name)
if not isinstance(driver, EventFiringWebDriver):
driver = EventFiringWebDriver(driver, event_listener())
request.node._driver = driver
yield driver
> driver.quit()
/usr/local/lib/python2.7/site-packages/pytest_selenium/pytest_selenium.py:143:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py:689: in quit
self.execute(Command.QUIT)
/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py:312: in execute
self.error_handler.check_response(response)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <selenium.webdriver.remote.errorhandler.ErrorHandler object at 0x7fde430d95d0>
response = {'status': 500, 'value': '{"value":{"error":"session not created","message":"Tried to run command without establishing...s::imp::thread::{{impl}}::new::thread_start\n at /checkout/src/libstd/sys/unix/thread.rs:84"}}'}
def check_response(self, response):
"""
Checks that a JSON response from the WebDriver does not have an error.
:Args:
- response - The JSON response from the WebDriver server as a dictionary
object.
:Raises: If the response contains an error message.
"""
status = response.get('status', None)
if status is None or status == ErrorCode.SUCCESS:
return
value = None
message = response.get("message", "")
screen = response.get("screen", "")
stacktrace = None
if isinstance(status, int):
value_json = response.get('value', None)
if value_json and isinstance(value_json, basestring):
import json
try:
value = json.loads(value_json)
if len(value.keys()) == 1:
value = value['value']
status = value.get('error', None)
if status is None:
status = value["status"]
message = value["value"]
if not isinstance(message, basestring):
value = message
message = message.get('message')
else:
message = value.get('message', None)
except ValueError:
pass
exception_class = ErrorInResponseException
if status in ErrorCode.NO_SUCH_ELEMENT:
exception_class = NoSuchElementException
elif status in ErrorCode.NO_SUCH_FRAME:
exception_class = NoSuchFrameException
elif status in ErrorCode.NO_SUCH_WINDOW:
exception_class = NoSuchWindowException
elif status in ErrorCode.STALE_ELEMENT_REFERENCE:
exception_class = StaleElementReferenceException
elif status in ErrorCode.ELEMENT_NOT_VISIBLE:
exception_class = ElementNotVisibleException
elif status in ErrorCode.INVALID_ELEMENT_STATE:
exception_class = InvalidElementStateException
elif status in ErrorCode.INVALID_SELECTOR \
or status in ErrorCode.INVALID_XPATH_SELECTOR \
or status in ErrorCode.INVALID_XPATH_SELECTOR_RETURN_TYPER:
exception_class = InvalidSelectorException
elif status in ErrorCode.ELEMENT_IS_NOT_SELECTABLE:
exception_class = ElementNotSelectableException
elif status in ErrorCode.ELEMENT_NOT_INTERACTABLE:
exception_class = ElementNotInteractableException
elif status in ErrorCode.INVALID_COOKIE_DOMAIN:
exception_class = InvalidCookieDomainException
elif status in ErrorCode.UNABLE_TO_SET_COOKIE:
exception_class = UnableToSetCookieException
elif status in ErrorCode.TIMEOUT:
exception_class = TimeoutException
elif status in ErrorCode.SCRIPT_TIMEOUT:
exception_class = TimeoutException
elif status in ErrorCode.UNKNOWN_ERROR:
exception_class = WebDriverException
elif status in ErrorCode.UNEXPECTED_ALERT_OPEN:
exception_class = UnexpectedAlertPresentException
elif status in ErrorCode.NO_ALERT_OPEN:
exception_class = NoAlertPresentException
elif status in ErrorCode.IME_NOT_AVAILABLE:
exception_class = ImeNotAvailableException
elif status in ErrorCode.IME_ENGINE_ACTIVATION_FAILED:
exception_class = ImeActivationFailedException
elif status in ErrorCode.MOVE_TARGET_OUT_OF_BOUNDS:
exception_class = MoveTargetOutOfBoundsException
elif status in ErrorCode.JAVASCRIPT_ERROR:
exception_class = JavascriptException
elif status in ErrorCode.SESSION_NOT_CREATED:
exception_class = SessionNotCreatedException
elif status in ErrorCode.INVALID_ARGUMENT:
exception_class = InvalidArgumentException
elif status in ErrorCode.NO_SUCH_COOKIE:
exception_class = NoSuchCookieException
elif status in ErrorCode.UNABLE_TO_CAPTURE_SCREEN:
exception_class = ScreenshotException
elif status in ErrorCode.ELEMENT_CLICK_INTERCEPTED:
exception_class = ElementClickInterceptedException
elif status in ErrorCode.INSECURE_CERTIFICATE:
exception_class = InsecureCertificateException
elif status in ErrorCode.INVALID_COORDINATES:
exception_class = InvalidCoordinatesException
elif status in ErrorCode.INVALID_SESSION_ID:
exception_class = InvalidSessionIdException
elif status in ErrorCode.UNKNOWN_METHOD:
exception_class = UnknownMethodException
else:
exception_class = WebDriverException
if value == '' or value is None:
value = response['value']
if isinstance(value, basestring):
if exception_class == ErrorInResponseException:
raise exception_class(response, value)
raise exception_class(value)
if message == "" and 'message' in value:
message = value['message']
screen = None
if 'screen' in value:
screen = value['screen']
stacktrace = None
if 'stackTrace' in value and value['stackTrace']:
stacktrace = []
try:
for frame in value['stackTrace']:
line = self._value_or_default(frame, 'lineNumber', '')
file = self._value_or_default(frame, 'fileName', '<anonymous>')
if line:
file = "%s:%s" % (file, line)
meth = self._value_or_default(frame, 'methodName', '<anonymous>')
if 'className' in frame:
meth = "%s.%s" % (frame['className'], meth)
msg = " at %s (%s)"
msg = msg % (meth, file)
stacktrace.append(msg)
except TypeError:
pass
if exception_class == ErrorInResponseException:
raise exception_class(response, message)
elif exception_class == UnexpectedAlertPresentException and 'alert' in value:
raise exception_class(message, screen, stacktrace, value['alert'].get('text'))
> raise exception_class(message, screen, stacktrace)
E SessionNotCreatedException: Message: Tried to run command without establishing a connection
/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py:237: SessionNotCreatedException
------------------------------------------------- pytest-selenium -------------------------------------------------
WARNING: Failed to gather URL: Message: No active session with ID 8b140dd6-30bd-4a3e-b3ba-9c8c56b6ec01
WARNING: Failed to gather screenshot: Message: No active session with ID 8b140dd6-30bd-4a3e-b3ba-9c8c56b6ec01
WARNING: Failed to gather HTML: Message: No active session with ID 8b140dd6-30bd-4a3e-b3ba-9c8c56b6ec01
WARNING: Failed to gather log types: Message:
==================================================== FAILURES =====================================================
__________________________________________ test_dashboard_load_page_two ___________________________________________
base_url = 'https://developer-prod.mdn.mozit.cloud'
selenium = <selenium.webdriver.remote.webdriver.WebDriver (session="8b140dd6-30bd-4a3e-b3ba-9c8c56b6ec01")>
@pytest.mark.smoke
@pytest.mark.nondestructive
def test_dashboard_load_page_two(base_url, selenium):
page = DashboardPage(selenium, base_url).open()
# save id of first revision on page one
first_row_id = page.first_row_id
# click on page two link
> page.click_page_two()
tests/functional/test_dashboard.py:50:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/pages/dashboard.py:107: in click_page_two
self.wait.until(lambda s: 'closed' in self.find_element(*self._first_notification_locator).get_attribute('class'))
/usr/local/lib/python2.7/site-packages/selenium/webdriver/support/wait.py:71: in until
value = method(self._driver)
tests/pages/dashboard.py:107: in <lambda>
self.wait.until(lambda s: 'closed' in self.find_element(*self._first_notification_locator).get_attribute('class'))
/usr/local/lib/python2.7/site-packages/pypom/view.py:32: in find_element
return self.selenium.find_element(strategy, locator)
/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py:955: in find_element
'value': value})['value']
/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py:312: in execute
self.error_handler.check_response(response)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <selenium.webdriver.remote.errorhandler.ErrorHandler object at 0x7fde430d95d0>
response = {'status': 500, 'value': '{"value":{"error":"unknown error","message":"Failed to decode response from marionette","sta...s::imp::thread::{{impl}}::new::thread_start\n at /checkout/src/libstd/sys/unix/thread.rs:84"}}'}
def check_response(self, response):
"""
Checks that a JSON response from the WebDriver does not have an error.
:Args:
- response - The JSON response from the WebDriver server as a dictionary
object.
:Raises: If the response contains an error message.
"""
status = response.get('status', None)
if status is None or status == ErrorCode.SUCCESS:
return
value = None
message = response.get("message", "")
screen = response.get("screen", "")
stacktrace = None
if isinstance(status, int):
value_json = response.get('value', None)
if value_json and isinstance(value_json, basestring):
import json
try:
value = json.loads(value_json)
if len(value.keys()) == 1:
value = value['value']
status = value.get('error', None)
if status is None:
status = value["status"]
message = value["value"]
if not isinstance(message, basestring):
value = message
message = message.get('message')
else:
message = value.get('message', None)
except ValueError:
pass
exception_class = ErrorInResponseException
if status in ErrorCode.NO_SUCH_ELEMENT:
exception_class = NoSuchElementException
elif status in ErrorCode.NO_SUCH_FRAME:
exception_class = NoSuchFrameException
elif status in ErrorCode.NO_SUCH_WINDOW:
exception_class = NoSuchWindowException
elif status in ErrorCode.STALE_ELEMENT_REFERENCE:
exception_class = StaleElementReferenceException
elif status in ErrorCode.ELEMENT_NOT_VISIBLE:
exception_class = ElementNotVisibleException
elif status in ErrorCode.INVALID_ELEMENT_STATE:
exception_class = InvalidElementStateException
elif status in ErrorCode.INVALID_SELECTOR \
or status in ErrorCode.INVALID_XPATH_SELECTOR \
or status in ErrorCode.INVALID_XPATH_SELECTOR_RETURN_TYPER:
exception_class = InvalidSelectorException
elif status in ErrorCode.ELEMENT_IS_NOT_SELECTABLE:
exception_class = ElementNotSelectableException
elif status in ErrorCode.ELEMENT_NOT_INTERACTABLE:
exception_class = ElementNotInteractableException
elif status in ErrorCode.INVALID_COOKIE_DOMAIN:
exception_class = InvalidCookieDomainException
elif status in ErrorCode.UNABLE_TO_SET_COOKIE:
exception_class = UnableToSetCookieException
elif status in ErrorCode.TIMEOUT:
exception_class = TimeoutException
elif status in ErrorCode.SCRIPT_TIMEOUT:
exception_class = TimeoutException
elif status in ErrorCode.UNKNOWN_ERROR:
exception_class = WebDriverException
elif status in ErrorCode.UNEXPECTED_ALERT_OPEN:
exception_class = UnexpectedAlertPresentException
elif status in ErrorCode.NO_ALERT_OPEN:
exception_class = NoAlertPresentException
elif status in ErrorCode.IME_NOT_AVAILABLE:
exception_class = ImeNotAvailableException
elif status in ErrorCode.IME_ENGINE_ACTIVATION_FAILED:
exception_class = ImeActivationFailedException
elif status in ErrorCode.MOVE_TARGET_OUT_OF_BOUNDS:
exception_class = MoveTargetOutOfBoundsException
elif status in ErrorCode.JAVASCRIPT_ERROR:
exception_class = JavascriptException
elif status in ErrorCode.SESSION_NOT_CREATED:
exception_class = SessionNotCreatedException
elif status in ErrorCode.INVALID_ARGUMENT:
exception_class = InvalidArgumentException
elif status in ErrorCode.NO_SUCH_COOKIE:
exception_class = NoSuchCookieException
elif status in ErrorCode.UNABLE_TO_CAPTURE_SCREEN:
exception_class = ScreenshotException
elif status in ErrorCode.ELEMENT_CLICK_INTERCEPTED:
exception_class = ElementClickInterceptedException
elif status in ErrorCode.INSECURE_CERTIFICATE:
exception_class = InsecureCertificateException
elif status in ErrorCode.INVALID_COORDINATES:
exception_class = InvalidCoordinatesException
elif status in ErrorCode.INVALID_SESSION_ID:
exception_class = InvalidSessionIdException
elif status in ErrorCode.UNKNOWN_METHOD:
exception_class = UnknownMethodException
else:
exception_class = WebDriverException
if value == '' or value is None:
value = response['value']
if isinstance(value, basestring):
if exception_class == ErrorInResponseException:
raise exception_class(response, value)
raise exception_class(value)
if message == "" and 'message' in value:
message = value['message']
screen = None
if 'screen' in value:
screen = value['screen']
stacktrace = None
if 'stackTrace' in value and value['stackTrace']:
stacktrace = []
try:
for frame in value['stackTrace']:
line = self._value_or_default(frame, 'lineNumber', '')
file = self._value_or_default(frame, 'fileName', '<anonymous>')
if line:
file = "%s:%s" % (file, line)
meth = self._value_or_default(frame, 'methodName', '<anonymous>')
if 'className' in frame:
meth = "%s.%s" % (frame['className'], meth)
msg = " at %s (%s)"
msg = msg % (meth, file)
stacktrace.append(msg)
except TypeError:
pass
if exception_class == ErrorInResponseException:
raise exception_class(response, message)
elif exception_class == UnexpectedAlertPresentException and 'alert' in value:
raise exception_class(message, screen, stacktrace, value['alert'].get('text'))
> raise exception_class(message, screen, stacktrace)
E WebDriverException: Message: Failed to decode response from marionette
/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py:237: WebDriverException
------------------------------------------------- pytest-selenium -------------------------------------------------
WARNING: Failed to gather URL: Message: Tried to run command without establishing a connection
WARNING: Failed to gather screenshot: Message: Tried to run command without establishing a connection
WARNING: Failed to gather HTML: Message: Tried to run command without establishing a connection
______________________________________________ test_search_homepage _______________________________________________
base_url = 'https://developer-prod.mdn.mozit.cloud'
selenium = <selenium.webdriver.remote.webdriver.WebDriver (session="30f85b9c-2bb2-4b1c-9b57-ee7509889650")>
@pytest.mark.smoke
@pytest.mark.search
@pytest.mark.nondestructive
def test_search_homepage(base_url, selenium):
# open homepage
page = HomePage(selenium, base_url).open()
# search for CSS in big box
search = page.search_for_term(SEARCH_TERM)
# search term is in search box
> assert search.search_input_value == SEARCH_TERM
tests/functional/test_search.py:23:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/pages/search.py:31: in search_input_value
return self.find_element(*self._search_input_locator).get_attribute('value')
/usr/local/lib/python2.7/site-packages/pypom/view.py:32: in find_element
return self.selenium.find_element(strategy, locator)
/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py:955: in find_element
'value': value})['value']
/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py:312: in execute
self.error_handler.check_response(response)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <selenium.webdriver.remote.errorhandler.ErrorHandler object at 0x7fde429f0910>
response = {'status': 404, 'value': '{"value":{"error":"no such element","message":"Unable to locate element: [id=\"search-q\"]",...Error@chrome://marionette/content/error.js:463:5\nelement.find/</<@chrome://marionette/content/element.js:291:16\n"}}'}
def check_response(self, response):
"""
Checks that a JSON response from the WebDriver does not have an error.
:Args:
- response - The JSON response from the WebDriver server as a dictionary
object.
:Raises: If the response contains an error message.
"""
status = response.get('status', None)
if status is None or status == ErrorCode.SUCCESS:
return
value = None
message = response.get("message", "")
screen = response.get("screen", "")
stacktrace = None
if isinstance(status, int):
value_json = response.get('value', None)
if value_json and isinstance(value_json, basestring):
import json
try:
value = json.loads(value_json)
if len(value.keys()) == 1:
value = value['value']
status = value.get('error', None)
if status is None:
status = value["status"]
message = value["value"]
if not isinstance(message, basestring):
value = message
message = message.get('message')
else:
message = value.get('message', None)
except ValueError:
pass
exception_class = ErrorInResponseException
if status in ErrorCode.NO_SUCH_ELEMENT:
exception_class = NoSuchElementException
elif status in ErrorCode.NO_SUCH_FRAME:
exception_class = NoSuchFrameException
elif status in ErrorCode.NO_SUCH_WINDOW:
exception_class = NoSuchWindowException
elif status in ErrorCode.STALE_ELEMENT_REFERENCE:
exception_class = StaleElementReferenceException
elif status in ErrorCode.ELEMENT_NOT_VISIBLE:
exception_class = ElementNotVisibleException
elif status in ErrorCode.INVALID_ELEMENT_STATE:
exception_class = InvalidElementStateException
elif status in ErrorCode.INVALID_SELECTOR \
or status in ErrorCode.INVALID_XPATH_SELECTOR \
or status in ErrorCode.INVALID_XPATH_SELECTOR_RETURN_TYPER:
exception_class = InvalidSelectorException
elif status in ErrorCode.ELEMENT_IS_NOT_SELECTABLE:
exception_class = ElementNotSelectableException
elif status in ErrorCode.ELEMENT_NOT_INTERACTABLE:
exception_class = ElementNotInteractableException
elif status in ErrorCode.INVALID_COOKIE_DOMAIN:
exception_class = InvalidCookieDomainException
elif status in ErrorCode.UNABLE_TO_SET_COOKIE:
exception_class = UnableToSetCookieException
elif status in ErrorCode.TIMEOUT:
exception_class = TimeoutException
elif status in ErrorCode.SCRIPT_TIMEOUT:
exception_class = TimeoutException
elif status in ErrorCode.UNKNOWN_ERROR:
exception_class = WebDriverException
elif status in ErrorCode.UNEXPECTED_ALERT_OPEN:
exception_class = UnexpectedAlertPresentException
elif status in ErrorCode.NO_ALERT_OPEN:
exception_class = NoAlertPresentException
elif status in ErrorCode.IME_NOT_AVAILABLE:
exception_class = ImeNotAvailableException
elif status in ErrorCode.IME_ENGINE_ACTIVATION_FAILED:
exception_class = ImeActivationFailedException
elif status in ErrorCode.MOVE_TARGET_OUT_OF_BOUNDS:
exception_class = MoveTargetOutOfBoundsException
elif status in ErrorCode.JAVASCRIPT_ERROR:
exception_class = JavascriptException
elif status in ErrorCode.SESSION_NOT_CREATED:
exception_class = SessionNotCreatedException
elif status in ErrorCode.INVALID_ARGUMENT:
exception_class = InvalidArgumentException
elif status in ErrorCode.NO_SUCH_COOKIE:
exception_class = NoSuchCookieException
elif status in ErrorCode.UNABLE_TO_CAPTURE_SCREEN:
exception_class = ScreenshotException
elif status in ErrorCode.ELEMENT_CLICK_INTERCEPTED:
exception_class = ElementClickInterceptedException
elif status in ErrorCode.INSECURE_CERTIFICATE:
exception_class = InsecureCertificateException
elif status in ErrorCode.INVALID_COORDINATES:
exception_class = InvalidCoordinatesException
elif status in ErrorCode.INVALID_SESSION_ID:
exception_class = InvalidSessionIdException
elif status in ErrorCode.UNKNOWN_METHOD:
exception_class = UnknownMethodException
else:
exception_class = WebDriverException
if value == '' or value is None:
value = response['value']
if isinstance(value, basestring):
if exception_class == ErrorInResponseException:
raise exception_class(response, value)
raise exception_class(value)
if message == "" and 'message' in value:
message = value['message']
screen = None
if 'screen' in value:
screen = value['screen']
stacktrace = None
if 'stackTrace' in value and value['stackTrace']:
stacktrace = []
try:
for frame in value['stackTrace']:
line = self._value_or_default(frame, 'lineNumber', '')
file = self._value_or_default(frame, 'fileName', '<anonymous>')
if line:
file = "%s:%s" % (file, line)
meth = self._value_or_default(frame, 'methodName', '<anonymous>')
if 'className' in frame:
meth = "%s.%s" % (frame['className'], meth)
msg = " at %s (%s)"
msg = msg % (meth, file)
stacktrace.append(msg)
except TypeError:
pass
if exception_class == ErrorInResponseException:
raise exception_class(response, message)
elif exception_class == UnexpectedAlertPresentException and 'alert' in value:
raise exception_class(message, screen, stacktrace, value['alert'].get('text'))
> raise exception_class(message, screen, stacktrace)
E NoSuchElementException: Message: Unable to locate element: [id="search-q"]
/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py:237: NoSuchElementException
------------------------------------------------- pytest-selenium -------------------------------------------------
URL: https://developer-prod.mdn.mozit.cloud/en-US/search?q=css&topic=apps&topic=html&topic=css&topic=js&topic=api&topic=canvas&topic=svg&topic=webgl&topic=mobile&topic=webdev&topic=http&topic=webext&topic=standards
=============================================== 7 tests deselected ================================================
=============== 2 failed, 103 passed, 6 skipped, 7 deselected, 1 xpassed, 1 error in 412.20 seconds ===============
*** Shutting down dockerized browsers...
selenium-chrome-kuma
selenium-chrome-kuma
selenium-firefox-kuma
selenium-firefox-kuma
*** Test results in test_results/functional_20180912_1501The Chrome tests passed, and the Firefox tests had two tests with issues:
tests/functional/test_dashboard.py::test_dashboard_load_page_two (this test generated an error as well as a failure in the run above)tests/functional/test_search.py::test_search_homepage Both of theses tests ran as expected when run separately.
Here are the details of the successful re-run (sucessful expected failure) of test_dashboard.py::test_dashboard_load_page_two:
(mdntest) rjohnson-25186:kuma rjohnson$ BROWSERS=firefox scripts/run_functional_tests.sh --maintenance-mode --base-url https://developer-prod.mdn.mozit.cloud -m "not login" tests/functional/test_dashboard.py::test_dashboard_load_page_two
*** Building integration tests image...
Sending build context to Docker daemon 179.3MB
Step 1/10 : FROM python:2.7-slim
2.7-slim: Pulling from library/python
Digest: sha256:0a43a6d7858af4a42427c792b682936d2cd34e183fb026627f53ddb556d4bf62
Status: Image is up to date for python:2.7-slim
---> c9cde4658340
Step 2/10 : WORKDIR /app
---> Using cache
---> fda1425b260f
Step 3/10 : RUN set -ex && apt-get update && apt-get install -y --no-install-recommends mime-support build-essential libxml2-dev libxslt1.1 libxslt1-dev zlib1g-dev && rm -rf /var/lib/apt/lists/*
---> Using cache
---> c7cd96772567
Step 4/10 : ENV PYTEST_PROCESSES 5
---> Using cache
---> 6a8ffee2f698
Step 5/10 : ENV PRIVACY "public restricted"
---> Using cache
---> 32ac9e9d4741
Step 6/10 : ENV TESTS_PATH /app/tests
---> Using cache
---> 43a2778303ec
Step 7/10 : ENV RESULTS_PATH /app/results
---> Using cache
---> 0c3166366abb
Step 8/10 : COPY ./requirements /app/requirements
---> Using cache
---> 4d3ee7eb9284
Step 9/10 : RUN pip install --no-cache-dir -r requirements/test.txt
---> Using cache
---> 29fc609364e0
Step 10/10 : COPY tests /app/tests
---> Using cache
---> 3976ba7b1070
Successfully built 3976ba7b1070
Successfully tagged kuma-integration-tests:latest
*** Running dockerized firefox...
50605d2230e5b07397949598b6a1b0b74ea4dfc8668c9b5b73cd16af603bf9cd
*** Running integration tests against firefox...
=============================================== test session starts ===============================================
platform linux2 -- Python 2.7.15, pytest-3.1.3, py-1.4.33, pluggy-0.4.0
driver: Remote
sensitiveurl: .* *** WARNING: sensitive url matches https://developer-prod.mdn.mozit.cloud ***
metadata: {'Python': '2.7.15', 'Driver': 'Remote', 'Capabilities': {'browserName': 'firefox'}, 'Server': 'hub:4444', 'Base URL': 'https://developer-prod.mdn.mozit.cloud', 'Platform': 'Linux-4.9.93-linuxkit-aufs-x86_64-with-debian-9.5', 'kuma': {u'services': {u'search': {u'available': True, u'count': 60165, u'populated': True}, u'kumascript': {u'available': True, u'revision': u'e57e27c371416c7e104f7843318e9166dbe3c1f3'}, u'test_accounts': {u'available': False}, u'database': {u'available': True, u'document_count': 130018, u'populated': True}}, u'version': 1, u'request': {u'url': u'https://developer-prod.mdn.mozit.cloud/_kuma_status.json', u'is_secure': True, u'host': u'developer-prod.mdn.mozit.cloud', u'scheme': u'https'}, 'response': {'headers': {'X-Cache': 'Miss from cloudfront', 'strict-transport-security': 'max-age=63072000', 'x-content-type-options': 'nosniff', 'Content-Language': 'en-US', 'Content-Encoding': 'gzip', 'Transfer-Encoding': 'chunked', 'Expires': 'Wed, 12 Sep 2018 22:53:46 GMT', 'Vary': 'Accept-Encoding', 'Server': 'meinheld/0.6.1', 'Connection': 'keep-alive', 'x-xss-protection': '1; mode=block', 'X-Amz-Cf-Id': 'TMTD5z3SQvjBpCM4HaztolgU5cHDWnu3nGAFsiV6lXdEhNhRRKL_5A==', 'Cache-Control': 'no-cache, no-store, must-revalidate, max-age=0', 'Date': 'Wed, 12 Sep 2018 22:53:46 GMT', 'X-Frame-Options': 'DENY', 'Content-Type': 'application/json', 'Via': '1.1 19f9923c4e449b92312c8813bf9135f5.cloudfront.net (CloudFront)'}}, u'settings': {u'ATTACHMENT_HOST': u'demos.mdn.mozit.cloud', u'PROTOCOL': u'https://', u'INTERACTIVE_EXAMPLES_BASE': u'https://interactive-examples.mdn.mozit.cloud', u'MAINTENANCE_MODE': True, u'STATIC_URL': u'https://developer-prod.mdn.mozit.cloud/static/', u'SITE_URL': u'https://developer-prod.mdn.mozit.cloud', u'ATTACHMENT_ORIGIN': u'prod.mdn.mozit.cloud', u'DEBUG': False, u'ALLOWED_HOSTS': [u'developer.mozilla.org', u'cdn.mdn.mozilla.net', u'mdn.mozillademos.org', u'demos.mdn.mozit.cloud', u'developer-prod.mdn.mozit.cloud', u'prod.mdn.mozit.cloud', u'developer.mozilla.com', u'mdn.mozilla.org', u'developer-new.mozilla.org', u'developers.mozilla.org'], u'REVISION_HASH': u'0591dd661bfa5e34f19a74428dff5cc8a172998c', u'LEGACY_HOSTS': [u'cdn.mdn.mozilla.net', u'developer.mozilla.com', u'mdn.mozilla.org', u'developer-new.mozilla.org', u'developers.mozilla.org']}}, 'Plugins': {'variables': '1.7.1', 'selenium': '1.11.4', 'rerunfailures': '2.1.0', 'html': '1.16.1', 'base-url': '1.4.1', 'metadata': '1.5.1'}, 'Packages': {'py': '1.4.33', 'pytest': '3.1.3', 'pluggy': '0.4.0'}}
baseurl: https://developer-prod.mdn.mozit.cloud
rootdir: /app, inifile:
plugins: selenium-1.11.4, html-1.16.1, rerunfailures-2.1.0, variables-1.7.1, metadata-1.5.1, base-url-1.4.1
collected 1 item
tests/functional/test_dashboard.py x
------------------------------------ generated html file: /results/pytest.html ------------------------------------
=========================================== 1 xfailed in 14.94 seconds ============================================
*** Shutting down dockerized browsers...
selenium-firefox-kuma
selenium-firefox-kuma
*** Test results in test_results/functional_20180912_1553Here are the details of the successful re-run of tests/functional/test_search.py::test_search_homepage:
(mdntest) rjohnson-25186:kuma rjohnson$ BROWSERS=firefox scripts/run_functional_tests.sh --maintenance-mode --base-url https://developer-prod.mdn.mozit.cloud -m "not login" tests/functional/test_search.py::test_search_homepage
*** Building integration tests image...
Sending build context to Docker daemon 179.3MB
Step 1/10 : FROM python:2.7-slim
2.7-slim: Pulling from library/python
Digest: sha256:0a43a6d7858af4a42427c792b682936d2cd34e183fb026627f53ddb556d4bf62
Status: Image is up to date for python:2.7-slim
---> c9cde4658340
Step 2/10 : WORKDIR /app
---> Using cache
---> fda1425b260f
Step 3/10 : RUN set -ex && apt-get update && apt-get install -y --no-install-recommends mime-support build-essential libxml2-dev libxslt1.1 libxslt1-dev zlib1g-dev && rm -rf /var/lib/apt/lists/*
---> Using cache
---> c7cd96772567
Step 4/10 : ENV PYTEST_PROCESSES 5
---> Using cache
---> 6a8ffee2f698
Step 5/10 : ENV PRIVACY "public restricted"
---> Using cache
---> 32ac9e9d4741
Step 6/10 : ENV TESTS_PATH /app/tests
---> Using cache
---> 43a2778303ec
Step 7/10 : ENV RESULTS_PATH /app/results
---> Using cache
---> 0c3166366abb
Step 8/10 : COPY ./requirements /app/requirements
---> Using cache
---> 4d3ee7eb9284
Step 9/10 : RUN pip install --no-cache-dir -r requirements/test.txt
---> Using cache
---> 29fc609364e0
Step 10/10 : COPY tests /app/tests
---> Using cache
---> 3976ba7b1070
Successfully built 3976ba7b1070
Successfully tagged kuma-integration-tests:latest
*** Running dockerized firefox...
b41f2f8da480ceb4bb6fcbcbe2cb908e26db390551036d22fd79a514fcdf2e54
*** Running integration tests against firefox...
=============================================== test session starts ===============================================
platform linux2 -- Python 2.7.15, pytest-3.1.3, py-1.4.33, pluggy-0.4.0
driver: Remote
sensitiveurl: .* *** WARNING: sensitive url matches https://developer-prod.mdn.mozit.cloud ***
metadata: {'Python': '2.7.15', 'Driver': 'Remote', 'Capabilities': {'browserName': 'firefox'}, 'Server': 'hub:4444', 'Base URL': 'https://developer-prod.mdn.mozit.cloud', 'Platform': 'Linux-4.9.93-linuxkit-aufs-x86_64-with-debian-9.5', 'kuma': {u'services': {u'search': {u'available': True, u'count': 60165, u'populated': True}, u'kumascript': {u'available': True, u'revision': u'e57e27c371416c7e104f7843318e9166dbe3c1f3'}, u'test_accounts': {u'available': False}, u'database': {u'available': True, u'document_count': 130018, u'populated': True}}, u'version': 1, u'request': {u'url': u'https://developer-prod.mdn.mozit.cloud/_kuma_status.json', u'is_secure': True, u'host': u'developer-prod.mdn.mozit.cloud', u'scheme': u'https'}, 'response': {'headers': {'X-Cache': 'Miss from cloudfront', 'strict-transport-security': 'max-age=63072000', 'x-content-type-options': 'nosniff', 'Content-Language': 'en-US', 'Content-Encoding': 'gzip', 'Transfer-Encoding': 'chunked', 'Expires': 'Wed, 12 Sep 2018 22:57:46 GMT', 'Vary': 'Accept-Encoding', 'Server': 'meinheld/0.6.1', 'Connection': 'keep-alive', 'x-xss-protection': '1; mode=block', 'X-Amz-Cf-Id': 'ozWfIbu0UDutRBQrDIlnWdIiz4sYNfnc2g6bbW5711rFlRxQA0oFZg==', 'Cache-Control': 'no-cache, no-store, must-revalidate, max-age=0', 'Date': 'Wed, 12 Sep 2018 22:57:46 GMT', 'X-Frame-Options': 'DENY', 'Content-Type': 'application/json', 'Via': '1.1 817705cae774a11f586612cbd427091c.cloudfront.net (CloudFront)'}}, u'settings': {u'ATTACHMENT_HOST': u'demos.mdn.mozit.cloud', u'PROTOCOL': u'https://', u'INTERACTIVE_EXAMPLES_BASE': u'https://interactive-examples.mdn.mozit.cloud', u'MAINTENANCE_MODE': True, u'STATIC_URL': u'https://developer-prod.mdn.mozit.cloud/static/', u'SITE_URL': u'https://developer-prod.mdn.mozit.cloud', u'ATTACHMENT_ORIGIN': u'prod.mdn.mozit.cloud', u'DEBUG': False, u'ALLOWED_HOSTS': [u'developer.mozilla.org', u'cdn.mdn.mozilla.net', u'mdn.mozillademos.org', u'demos.mdn.mozit.cloud', u'developer-prod.mdn.mozit.cloud', u'prod.mdn.mozit.cloud', u'developer.mozilla.com', u'mdn.mozilla.org', u'developer-new.mozilla.org', u'developers.mozilla.org'], u'REVISION_HASH': u'0591dd661bfa5e34f19a74428dff5cc8a172998c', u'LEGACY_HOSTS': [u'cdn.mdn.mozilla.net', u'developer.mozilla.com', u'mdn.mozilla.org', u'developer-new.mozilla.org', u'developers.mozilla.org']}}, 'Plugins': {'variables': '1.7.1', 'selenium': '1.11.4', 'rerunfailures': '2.1.0', 'html': '1.16.1', 'base-url': '1.4.1', 'metadata': '1.5.1'}, 'Packages': {'py': '1.4.33', 'pytest': '3.1.3', 'pluggy': '0.4.0'}}
baseurl: https://developer-prod.mdn.mozit.cloud
rootdir: /app, inifile:
plugins: selenium-1.11.4, html-1.16.1, rerunfailures-2.1.0, variables-1.7.1, metadata-1.5.1, base-url-1.4.1
collected 1 item
tests/functional/test_search.py .
------------------------------------ generated html file: /results/pytest.html ------------------------------------
============================================ 1 passed in 5.36 seconds =============================================
*** Shutting down dockerized browsers...
selenium-firefox-kuma
selenium-firefox-kuma
*** Test results in test_results/functional_20180912_1557Headless tests require a Python environment with the requirements in
requirements/test.txt. See Setting up test virtual environment.
Run the headless tests, which make safe HTTP requests (no POSTs, no logins):
pytest --maintenance-mode --base-url https://developer-prod.mdn.mozit.cloud tests/headless The headless tests returned 4 failures, but each of the 4 failures is expected (returns a 302 status when a status of 200 was expected) since https://developer-prod.mdn.mozit.cloud is configured in maintenance mode. In other words, the failures are due to the tests not being properly designed to run correctly against a system in maintenance mode.
Details:
(mdntest) rjohnson-25186:kuma rjohnson$ pytest --maintenance-mode --base-url https://developer-prod.mdn.mozit.cloud tests/headless
=============================================== test session starts ===============================================
platform darwin -- Python 2.7.12, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
sensitiveurl: .* *** WARNING: sensitive url matches https://developer-prod.mdn.mozit.cloud ***
metadata: {'Python': '2.7.12', 'Driver': None, 'Capabilities': {}, 'Base URL': 'https://developer-prod.mdn.mozit.cloud', 'Platform': 'Darwin-17.7.0-x86_64-i386-64bit', 'kuma': {u'services': {u'search': {u'available': True, u'count': 60165, u'populated': True}, u'kumascript': {u'available': True, u'revision': u'e57e27c371416c7e104f7843318e9166dbe3c1f3'}, u'test_accounts': {u'available': False}, u'database': {u'available': True, u'document_count': 130018, u'populated': True}}, u'version': 1, u'request': {u'url': u'https://developer-prod.mdn.mozit.cloud/_kuma_status.json', u'is_secure': True, u'host': u'developer-prod.mdn.mozit.cloud', u'scheme': u'https'}, 'response': {'headers': {'X-Cache': 'Miss from cloudfront', 'strict-transport-security': 'max-age=63072000', 'x-content-type-options': 'nosniff', 'Content-Language': 'en-US', 'Content-Encoding': 'gzip', 'Transfer-Encoding': 'chunked', 'Expires': 'Wed, 12 Sep 2018 23:07:57 GMT', 'Vary': 'Accept-Encoding', 'Server': 'meinheld/0.6.1', 'Connection': 'keep-alive', 'x-xss-protection': '1; mode=block', 'X-Amz-Cf-Id': 'Eqp1NjIX9v548Mb97JR7LZNFqsm3qOUwb39izsudBgscKzTdCg1toQ==', 'Cache-Control': 'no-cache, no-store, must-revalidate, max-age=0', 'Date': 'Wed, 12 Sep 2018 23:07:57 GMT', 'X-Frame-Options': 'DENY', 'Content-Type': 'application/json', 'Via': '1.1 617383234aa18e133ce6e5179e83aa88.cloudfront.net (CloudFront)'}}, u'settings': {u'ATTACHMENT_HOST': u'demos.mdn.mozit.cloud', u'PROTOCOL': u'https://', u'INTERACTIVE_EXAMPLES_BASE': u'https://interactive-examples.mdn.mozit.cloud', u'MAINTENANCE_MODE': True, u'STATIC_URL': u'https://developer-prod.mdn.mozit.cloud/static/', u'SITE_URL': u'https://developer-prod.mdn.mozit.cloud', u'ATTACHMENT_ORIGIN': u'prod.mdn.mozit.cloud', u'DEBUG': False, u'ALLOWED_HOSTS': [u'developer.mozilla.org', u'cdn.mdn.mozilla.net', u'mdn.mozillademos.org', u'demos.mdn.mozit.cloud', u'developer-prod.mdn.mozit.cloud', u'prod.mdn.mozit.cloud', u'developer.mozilla.com', u'mdn.mozilla.org', u'developer-new.mozilla.org', u'developers.mozilla.org'], u'REVISION_HASH': u'0591dd661bfa5e34f19a74428dff5cc8a172998c', u'LEGACY_HOSTS': [u'cdn.mdn.mozilla.net', u'developer.mozilla.com', u'mdn.mozilla.org', u'developer-new.mozilla.org', u'developers.mozilla.org']}}, 'Plugins': {'variables': '1.7.1', 'selenium': '1.11.4', 'xdist': '1.16.0', 'rerunfailures': '2.1.0', 'html': '1.16.1', 'base-url': '1.4.1', 'metadata': '1.5.1'}, 'Packages': {'py': '1.5.3', 'pytest': '3.5.1', 'pluggy': '0.6.0'}}
baseurl: https://developer-prod.mdn.mozit.cloud
rootdir: /Users/rjohnson/repos/kuma, inifile: pytest.ini
plugins: xdist-1.16.0, variables-1.7.1, selenium-1.11.4, rerunfailures-2.1.0, metadata-1.5.1, html-1.16.1, base-url-1.4.1
collected 1696 items
tests/headless/test_cdn.py ...F.FF..................F.....................................................................s.................................................................................................................................................................................................................................................................................................................. [ 0%]
tests/headless/test_endpoints.py ... [ 0%]
tests/headless/test_redirects.py ...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... [ 0%]
tests/headless/test_robots.py .....
==================================================== FAILURES =====================================================
______________________________________ test_not_cached[/en-US/users/signin] _______________________________________
Traceback (most recent call last):
File "/Users/rjohnson/repos/kuma/tests/headless/test_cdn.py", line 117, in test_not_cached
assert_not_cached(base_url + slug, 200, is_behind_cdn)
File "/Users/rjohnson/repos/kuma/tests/headless/test_cdn.py", line 64, in assert_not_cached
**request_kwargs)
File "/Users/rjohnson/repos/kuma/tests/headless/test_cdn.py", line 52, in assert_not_cached_by_cdn
assert response.status_code == expected_status_code
AssertionError: assert 302 == 200
+ where 302 = <Response [302]>.status_code
______________________________________ test_not_cached[/en-US/unsubscribe/1] ______________________________________
Traceback (most recent call last):
File "/Users/rjohnson/repos/kuma/tests/headless/test_cdn.py", line 117, in test_not_cached
assert_not_cached(base_url + slug, 200, is_behind_cdn)
File "/Users/rjohnson/repos/kuma/tests/headless/test_cdn.py", line 64, in assert_not_cached
**request_kwargs)
File "/Users/rjohnson/repos/kuma/tests/headless/test_cdn.py", line 52, in assert_not_cached_by_cdn
assert response.status_code == expected_status_code
AssertionError: assert 302 == 200
+ where 302 = <Response [302]>.status_code
_________________________________________ test_not_cached[/admin/login/] __________________________________________
Traceback (most recent call last):
File "/Users/rjohnson/repos/kuma/tests/headless/test_cdn.py", line 117, in test_not_cached
assert_not_cached(base_url + slug, 200, is_behind_cdn)
File "/Users/rjohnson/repos/kuma/tests/headless/test_cdn.py", line 64, in assert_not_cached
**request_kwargs)
File "/Users/rjohnson/repos/kuma/tests/headless/test_cdn.py", line 52, in assert_not_cached_by_cdn
assert response.status_code == expected_status_code
AssertionError: assert 302 == 200
+ where 302 = <Response [302]>.status_code
___________________________ test_not_cached_admin_login_required[/admin/users/user/1/] ____________________________
Traceback (most recent call last):
File "/Users/rjohnson/repos/kuma/tests/headless/test_cdn.py", line 165, in test_not_cached_admin_login_required
assert response.headers['location'].endswith(
AssertionError: assert False
+ where False = <built-in method endswith of str object at 0x109767130>(('/admin/login/?next=' + '/admin/users/user/1/'))
+ where <built-in method endswith of str object at 0x109767130> = '/en-US/maintenance-mode'.endswith
+ and '/admin/users/user/1/' = quote('/admin/users/user/1/')
============================================= short test summary info =============================================
SKIP [1] tests/headless/test_cdn.py:295: unconditional skip
=============================== 4 failed, 1691 passed, 1 skipped in 398.56 seconds ================================Test these URLs as an anonymous user:
The following tests have been skipped for now as they do not apply when the system is in maintenance mode.
Regular Account Tests
Some things to try with a regular account, to exercise write functionality:
- [ ] Create a new MDN user account (may require deleting your
SocialAccount)- [ ] Send a account recovery link for an existing MDN user account
- [ ] Update the account profile
- [ ] Add and verify a new email for a profile
- [ ] Create a new page, such as https://developer-prod.mdn.mozit.cloud/en-US/docs/User:Test
- [ ] Create a translation of the new page
- [ ] Subscribe to a page, and change it with a different account
- [ ] Update the original page with a KumaScript macro, such as
{{cssxref("background")}}.- [ ] Update the translation of a changed English page
- [ ] Upload an image to the page
- [ ] Add the image to the page content
- [ ] Edit a zoned URL like https://developer-prod.mdn.mozit.cloud/en-US/Firefox/Releases
- [ ] Log out
Admin Tests
Some things to try with an admin account, to exercise restricted functionality:
- [ ] Move a page
- [ ] Delete a page
- [ ] Move a zoned page
- [ ] View https://developer-prod.mdn.mozit.cloud/admin/users/user/1/ - 200, first user
- [ ] View https://developer-prod.mdn.mozit.cloud/en-US/dashboards/spam - 200, "please wait" or dashboard
escattone
commented
6 years ago ✅ I've completed the checklist for maintenance mode. 🎉 All tests passed.
escattone
commented
6 years ago Before I close this, I'd like to do a few more things:
interactive-examples.mdn.mozilla.net domain.escattone
commented
6 years ago These tests were manually run to ensure that interactive examples run properly from their new prod infrastructure:
interactive-examples.mdn.mozit.cloud --> new interactive-examples CDN --> mdninteractive-b77d14bceaaa9ea4.s3-website-us-west-2.amazonaws.com (the website run from the mdninteractive-b77d14bceaaa9ea4 S3 bucket)These tests depend on https://github.com/mdn/infra/pull/71.
Prior to testing the endpoints listed below, I did the following:
https://github.com/mdn/infra/blob/master/apps/mdn/mdn-aws/k8s/regions/oregon/prod.mm.min.sh but with KUMA_MAINTENANCE_MODE set to False)The test performed for each endpoint listed below comprised the following steps:
shift-reload the page to re-render its EmbedInteractiveExample macro so that it uses interactive-examples.mdn.mozit.cloud rather than interactive-examples.mdn.mozilla.netinteractive-examples.mdn.mozit.cloudThe endpoints tested:
escattone
commented
6 years ago Created https://github.com/mozilla/mdn-k8s-private/pull/68 to add the New Relic application ID's and the SpeedCurve site ID to the mdn-secrets for the mdn-prod namespace within the oregon cluster.
escattone
commented
6 years ago Prior to performing these regular account tests, the site was placed into normal (read/write) mode.
Some things to try with a regular account, to exercise write functionality:
SocialAccount){{cssxref("background")}}.Some things to try with an admin account, to exercise restricted functionality:
escattone
commented
6 years ago Initial list of follow-on issues required prior to go-live:
escattone
commented
6 years ago This is complete. 🎉
Once stage is up and running setup the prod instance as a read-only, proof-of concept deployment.
Acceptance Criteria
Tasks:
oregon/prod.mm.sh(production read-only) andoregon/prod.mm.min.sh(production read-only but with stage-size replica counts)mdn-secrets.prodmdn-backup-secrets,mdn-newrelic-secrets,mdn-speedcurve-secrets.yaml, andmdn-secrets.prodmake k8s-sync-from-s3-cron) to pull (hourly) the EFS assets from the IT-owned S3 bucket into the IT-owned EFS volumeprod.mdn.mozit.cloudthat points to the new ELB for thewebservice created abovedeveloper-prod.mdn.mozit.cloudthat points to the new prod primary CDNdemos.mdn.mozit.cloudthat points to the new prod attachments CDNinteractive-examples.mdn.mozit.cloudthat points to the new interactive-examples CDNwebELB, the prod primary CDN, the attachments CDN, and the interactive-examples CDN (see comment below)developer-prod.mdn.mozit.cloudhumansandsitemaps(run./manage.py make_humansand./manage.py make_sitemapson the web pod)