escattone commented 6 years ago

Setup the standby (disaster-recovery) instance as a read-only (maintenance-mode) deployment.

User story

As an MDN developer, I want to ensure we have an instance of MDN available in another AWS region as a backup in case the production instance in the Oregon region experiences a catastrophic failure.

Acceptance Criteria

[x] Read-only instance of MDN running in Frankfurt-based Kubernetes cluster
[x] Automated tests / checklist run against this instance

Tasks:

[x] Create germany/prod.mm.sh (standby read-only) configuration
[x] Make the K8s namespace, shared-storage, and services
[x] Add the access URL's of the backing services (RDS, Redis) to mdn-secrets.prod
[x] Deploy the mdn-backup-secrets, mdn-newrelic-secrets, mdn-speedcurve-secrets.yaml, and mdn-secrets.prod
[x] Deploy the S3 EFS sync (make k8s-restore-cron) to pull (hourly) the EFS assets from the IT-owned S3 bucket into the IT-owned EFS volume
[x] Create DNS CNAME for standby.mdn.mozit.cloud that points to the new ELB for the web service created above
[x] Make the K8s deployments
[x] Request increase in the domain name limit on ACM certs (from 10 to 12) within the eu-central-1 region so I can add standby.mdn.mozit.cloud to the ELB cert
[x] Create new standby ELB cert
[x] Perform checklist

escattone commented 6 years ago

This checklist of tests uses these URLs:

Base: https://standby.mdn.mozit.cloud
CDN: None
Demo: https://mdn.mozillademos.org
Interactive Examples: https://interactive-examples.mdn.mozilla.net

The standby instance

The standby instance uses a read-replica of the production RDS instance, the same Elasticsearch instance as production, and its own Redis and memcached instances.

Manual Sanity Check

Home page

Load https://standby.mdn.mozit.cloud/en-US

[x] Loads without errors
[x] Has the banner "MDN is currently in read-only maintenance mode. Learn more."
[x] Does not have "Sign in" or "View Profile" at the top
[x] Has entries for the Hacks Blog

Article page

Load https://standby.mdn.mozit.cloud/en-US/docs/Web/HTML

[x] Loads without errors
[x] Has Maintenance Mode banner
[x] Does not have "Sign in" or "View Profile" at the top

escattone commented 6 years ago

Functional Tests

With a Kuma environment and Docker, run the functional tests:

scripts/run_functional_tests.sh --maintenance-mode --base-url https://standby.mdn.mozit.cloud -m "not login" tests/functional

This will run a selection of safe Selenium tests (no login, no page edits) with Chrome and Firefox, running in Docker images. A failing test is automatically re-run before calling it a failure. Test results are created, with screenshots, tracebacks, and logs for failing tests, including xfail and rerun tests.

[x] Functional tests pass

(mdntest) rjohnson-25186:kuma rjohnson$ scripts/run_functional_tests.sh --maintenance-mode --base-url https://standby.mdn.mozit.cloud -m "not login" tests/functional
*** Building integration tests image...
Sending build context to Docker daemon  179.1MB
Step 1/10 : FROM python:2.7-slim
2.7-slim: Pulling from library/python
Digest: sha256:0a43a6d7858af4a42427c792b682936d2cd34e183fb026627f53ddb556d4bf62
Status: Image is up to date for python:2.7-slim
 ---> c9cde4658340
Step 2/10 : WORKDIR /app
 ---> Using cache
 ---> fda1425b260f
Step 3/10 : RUN set -ex &&     apt-get update &&     apt-get install -y --no-install-recommends         mime-support         build-essential         libxml2-dev         libxslt1.1         libxslt1-dev         zlib1g-dev     && rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> c7cd96772567
Step 4/10 : ENV PYTEST_PROCESSES 5
 ---> Using cache
 ---> 6a8ffee2f698
Step 5/10 : ENV PRIVACY "public restricted"
 ---> Using cache
 ---> 32ac9e9d4741
Step 6/10 : ENV TESTS_PATH /app/tests
 ---> Using cache
 ---> 43a2778303ec
Step 7/10 : ENV RESULTS_PATH /app/results
 ---> Using cache
 ---> 0c3166366abb
Step 8/10 : COPY ./requirements /app/requirements
 ---> Using cache
 ---> 4a070e4b6e93
Step 9/10 : RUN pip install --no-cache-dir -r requirements/test.txt
 ---> Using cache
 ---> 146be0d9612d
Step 10/10 : COPY tests /app/tests
 ---> Using cache
 ---> 6d4aea82bed8
Successfully built 6d4aea82bed8
Successfully tagged kuma-integration-tests:latest
*** Running dockerized chrome...
55243074e6bc6230714f2c8939406098f1cc701f9af269b60830900a851b61fd
*** Running dockerized firefox...
9f36503fdaaee2e23e0bc854ce29a428dd6b70f3ef8bf534ba857565cbba2a9c
*** Running integration tests against chrome...
=============================================== test session starts ===============================================
platform linux2 -- Python 2.7.15, pytest-3.1.3, py-1.4.33, pluggy-0.4.0
driver: Remote
sensitiveurl: .* *** WARNING: sensitive url matches https://standby.mdn.mozit.cloud ***
metadata: {'Python': '2.7.15', 'Driver': 'Remote', 'Capabilities': {'browserName': 'chrome'}, 'Server': 'hub:4444', 'Base URL': 'https://standby.mdn.mozit.cloud', 'Platform': 'Linux-4.9.93-linuxkit-aufs-x86_64-with-debian-9.5', 'kuma': {u'services': {u'search': {u'available': True, u'count': 60179, u'populated': True}, u'kumascript': {u'available': True, u'revision': u'725538d72c9713589a89fca9f8c771da59c396a0'}, u'test_accounts': {u'available': False}, u'database': {u'available': True, u'document_count': 130020, u'populated': True}}, u'version': 1, u'request': {u'url': u'https://standby.mdn.mozit.cloud/_kuma_status.json', u'is_secure': True, u'host': u'standby.mdn.mozit.cloud', u'scheme': u'https'}, 'response': {'headers': {'Content-Length': '1294', 'strict-transport-security': 'max-age=63072000', 'x-content-type-options': 'nosniff', 'Content-Language': 'en-US', 'Expires': 'Wed, 03 Oct 2018 22:42:30 GMT', 'Server': 'meinheld/0.6.1', 'Connection': 'keep-alive', 'x-xss-protection': '1; mode=block', 'Cache-Control': 'no-cache, no-store, must-revalidate, max-age=0', 'Date': 'Wed, 03 Oct 2018 22:42:29 GMT', 'X-Frame-Options': 'DENY', 'Content-Type': 'application/json'}}, u'settings': {u'ATTACHMENT_HOST': u'mdn.mozillademos.org', u'PROTOCOL': u'https://', u'INTERACTIVE_EXAMPLES_BASE': u'https://interactive-examples.mdn.mozilla.net', u'MAINTENANCE_MODE': True, u'STATIC_URL': u'https://developer.mozilla.org/static/', u'SITE_URL': u'https://developer.mozilla.org', u'ATTACHMENT_ORIGIN': u'prod.mdn.mozit.cloud', u'DEBUG': False, u'ALLOWED_HOSTS': [u'developer.mozilla.org', u'cdn.mdn.mozilla.net', u'mdn.mozillademos.org', u'demos.mdn.mozit.cloud', u'developer-prod.mdn.mozit.cloud', u'prod.mdn.mozit.cloud', u'standby.mdn.mozit.cloud', u'developer.mozilla.com', u'mdn.mozilla.org', u'developer-new.mozilla.org', u'developers.mozilla.org'], u'REVISION_HASH': u'135eceb4e68afa16125fbf5bff77349925a3c87d', u'LEGACY_HOSTS': [u'cdn.mdn.mozilla.net', u'developer.mozilla.com', u'mdn.mozilla.org', u'developer-new.mozilla.org', u'developers.mozilla.org']}}, 'Plugins': {'variables': '1.7.1', 'selenium': '1.11.4', 'rerunfailures': '2.1.0', 'html': '1.16.1', 'base-url': '1.4.1', 'metadata': '1.5.1'}, 'Packages': {'py': '1.4.33', 'pytest': '3.1.3', 'pluggy': '0.4.0'}}
baseurl: https://standby.mdn.mozit.cloud
rootdir: /app, inifile:
plugins: selenium-1.11.4, html-1.16.1, rerunfailures-2.1.0, variables-1.7.1, metadata-1.5.1, base-url-1.4.1
collected 119 items

tests/functional/test_article.py ..s..s...
tests/functional/test_article_edit.py s
tests/functional/test_article_revision.py ..
tests/functional/test_content_experiment.py ss
tests/functional/test_dashboard.py ...X......
tests/functional/test_feedback.py ...
tests/functional/test_home.py .....s..
tests/functional/test_language_selector.py .
tests/functional/test_maintenance_mode_redirects.py ...............................................................
tests/functional/test_notfound.py ...
tests/functional/test_profiles.py .
tests/functional/test_report.py ..
tests/functional/test_search.py .......

------------------------------------ generated html file: /results/pytest.html ------------------------------------
=============================================== 7 tests deselected ================================================
======================== 105 passed, 6 skipped, 7 deselected, 1 xpassed in 352.68 seconds =========================
*** Running integration tests against firefox...
=============================================== test session starts ===============================================
platform linux2 -- Python 2.7.15, pytest-3.1.3, py-1.4.33, pluggy-0.4.0
driver: Remote
sensitiveurl: .* *** WARNING: sensitive url matches https://standby.mdn.mozit.cloud ***
metadata: {'Python': '2.7.15', 'Driver': 'Remote', 'Capabilities': {'browserName': 'firefox'}, 'Server': 'hub:4444', 'Base URL': 'https://standby.mdn.mozit.cloud', 'Platform': 'Linux-4.9.93-linuxkit-aufs-x86_64-with-debian-9.5', 'kuma': {u'services': {u'search': {u'available': True, u'count': 60179, u'populated': True}, u'kumascript': {u'available': True, u'revision': u'725538d72c9713589a89fca9f8c771da59c396a0'}, u'test_accounts': {u'available': False}, u'database': {u'available': True, u'document_count': 130020, u'populated': True}}, u'version': 1, u'request': {u'url': u'https://standby.mdn.mozit.cloud/_kuma_status.json', u'is_secure': True, u'host': u'standby.mdn.mozit.cloud', u'scheme': u'https'}, 'response': {'headers': {'Content-Length': '1294', 'strict-transport-security': 'max-age=63072000', 'x-content-type-options': 'nosniff', 'Content-Language': 'en-US', 'Expires': 'Wed, 03 Oct 2018 22:48:25 GMT', 'Server': 'meinheld/0.6.1', 'Connection': 'keep-alive', 'x-xss-protection': '1; mode=block', 'Cache-Control': 'no-cache, no-store, must-revalidate, max-age=0', 'Date': 'Wed, 03 Oct 2018 22:48:25 GMT', 'X-Frame-Options': 'DENY', 'Content-Type': 'application/json'}}, u'settings': {u'ATTACHMENT_HOST': u'mdn.mozillademos.org', u'PROTOCOL': u'https://', u'INTERACTIVE_EXAMPLES_BASE': u'https://interactive-examples.mdn.mozilla.net', u'MAINTENANCE_MODE': True, u'STATIC_URL': u'https://developer.mozilla.org/static/', u'SITE_URL': u'https://developer.mozilla.org', u'ATTACHMENT_ORIGIN': u'prod.mdn.mozit.cloud', u'DEBUG': False, u'ALLOWED_HOSTS': [u'developer.mozilla.org', u'cdn.mdn.mozilla.net', u'mdn.mozillademos.org', u'demos.mdn.mozit.cloud', u'developer-prod.mdn.mozit.cloud', u'prod.mdn.mozit.cloud', u'standby.mdn.mozit.cloud', u'developer.mozilla.com', u'mdn.mozilla.org', u'developer-new.mozilla.org', u'developers.mozilla.org'], u'REVISION_HASH': u'135eceb4e68afa16125fbf5bff77349925a3c87d', u'LEGACY_HOSTS': [u'cdn.mdn.mozilla.net', u'developer.mozilla.com', u'mdn.mozilla.org', u'developer-new.mozilla.org', u'developers.mozilla.org']}}, 'Plugins': {'variables': '1.7.1', 'selenium': '1.11.4', 'rerunfailures': '2.1.0', 'html': '1.16.1', 'base-url': '1.4.1', 'metadata': '1.5.1'}, 'Packages': {'py': '1.4.33', 'pytest': '3.1.3', 'pluggy': '0.4.0'}}
baseurl: https://standby.mdn.mozit.cloud
rootdir: /app, inifile:
plugins: selenium-1.11.4, html-1.16.1, rerunfailures-2.1.0, variables-1.7.1, metadata-1.5.1, base-url-1.4.1
collected 119 items

tests/functional/test_article.py ..s..s...
tests/functional/test_article_edit.py s
tests/functional/test_article_revision.py ..
tests/functional/test_content_experiment.py ss
tests/functional/test_dashboard.py ...X......
tests/functional/test_feedback.py ...
tests/functional/test_home.py .....s..
tests/functional/test_language_selector.py .
tests/functional/test_maintenance_mode_redirects.py ...............................................................
tests/functional/test_notfound.py ...
tests/functional/test_profiles.py .
tests/functional/test_report.py ..
tests/functional/test_search.py .RFR.....

------------------------------------ generated html file: /results/pytest.html ------------------------------------
==================================================== FAILURES =====================================================
_____________________________________________ test_search_home_header _____________________________________________

base_url = 'https://standby.mdn.mozit.cloud'
selenium = <selenium.webdriver.remote.webdriver.WebDriver (session="dc62600b-7e27-46eb-bf25-bf674d5e789f")>

    @pytest.mark.flaky(reruns=1)
    @pytest.mark.smoke
    @pytest.mark.search
    @pytest.mark.nondestructive
    def test_search_home_header(base_url, selenium):
        # open homepage
        page = HomePage(selenium, base_url).open()
        # focus on search in nav menu
        width_before = page.header.search_wrapper_width
        page.header.search_field_focus()
        width_after = page.header.search_wrapper_width
        assert width_before < width_after
        # search for CSS
        search = page.header.search_for_term(SEARCH_TERM)
        # search term is in search box
        assert search.search_input_value == SEARCH_TERM
        # results found
>       assert search.search_result_items_length == 10
E       assert 3 == 10
E        +  where 3 = <pages.search.SearchPage object at 0x7f1a23e22510>.search_result_items_length

tests/functional/test_search.py:45: AssertionError
------------------------------------------------- pytest-selenium -------------------------------------------------
URL: https://standby.mdn.mozit.cloud/en-US/search?q=css
=============================================== 7 tests deselected ================================================
=============== 1 failed, 104 passed, 6 skipped, 7 deselected, 1 xpassed, 2 rerun in 568.21 seconds ===============
*** Shutting down dockerized browsers...
selenium-chrome-kuma
selenium-chrome-kuma
selenium-firefox-kuma
selenium-firefox-kuma
*** Test results in test_results/functional_20181003_1542

Some of the Firefox search tests are flaky, so for example the test that failed above runs below (but with another test failing that passed above):

*** Running dockerized firefox...
14a61a47187ad122a491733cc921d241e2fb3532f4599360aa3a172c17616bf3
*** Running integration tests against firefox...
=============================================== test session starts ===============================================
platform linux2 -- Python 2.7.15, pytest-3.1.3, py-1.4.33, pluggy-0.4.0
driver: Remote
sensitiveurl: .* *** WARNING: sensitive url matches https://standby.mdn.mozit.cloud ***
metadata: {'Python': '2.7.15', 'Driver': 'Remote', 'Capabilities': {'browserName': 'firefox'}, 'Server': 'hub:4444', 'Base URL': 'https://standby.mdn.mozit.cloud', 'Platform': 'Linux-4.9.93-linuxkit-aufs-x86_64-with-debian-9.5', 'kuma': {u'services': {u'search': {u'available': True, u'count': 60179, u'populated': True}, u'kumascript': {u'available': True, u'revision': u'725538d72c9713589a89fca9f8c771da59c396a0'}, u'test_accounts': {u'available': False}, u'database': {u'available': True, u'document_count': 130020, u'populated': True}}, u'version': 1, u'request': {u'url': u'https://standby.mdn.mozit.cloud/_kuma_status.json', u'is_secure': True, u'host': u'standby.mdn.mozit.cloud', u'scheme': u'https'}, 'response': {'headers': {'Content-Length': '1294', 'strict-transport-security': 'max-age=63072000', 'x-content-type-options': 'nosniff', 'Content-Language': 'en-US', 'Expires': 'Wed, 03 Oct 2018 23:28:08 GMT', 'Server': 'meinheld/0.6.1', 'Connection': 'keep-alive', 'x-xss-protection': '1; mode=block', 'Cache-Control': 'no-cache, no-store, must-revalidate, max-age=0', 'Date': 'Wed, 03 Oct 2018 23:28:07 GMT', 'X-Frame-Options': 'DENY', 'Content-Type': 'application/json'}}, u'settings': {u'ATTACHMENT_HOST': u'mdn.mozillademos.org', u'PROTOCOL': u'https://', u'INTERACTIVE_EXAMPLES_BASE': u'https://interactive-examples.mdn.mozilla.net', u'MAINTENANCE_MODE': True, u'STATIC_URL': u'https://developer.mozilla.org/static/', u'SITE_URL': u'https://developer.mozilla.org', u'ATTACHMENT_ORIGIN': u'prod.mdn.mozit.cloud', u'DEBUG': False, u'ALLOWED_HOSTS': [u'developer.mozilla.org', u'cdn.mdn.mozilla.net', u'mdn.mozillademos.org', u'demos.mdn.mozit.cloud', u'developer-prod.mdn.mozit.cloud', u'prod.mdn.mozit.cloud', u'standby.mdn.mozit.cloud', u'developer.mozilla.com', u'mdn.mozilla.org', u'developer-new.mozilla.org', u'developers.mozilla.org'], u'REVISION_HASH': u'135eceb4e68afa16125fbf5bff77349925a3c87d', u'LEGACY_HOSTS': [u'cdn.mdn.mozilla.net', u'developer.mozilla.com', u'mdn.mozilla.org', u'developer-new.mozilla.org', u'developers.mozilla.org']}}, 'Plugins': {'variables': '1.7.1', 'selenium': '1.11.4', 'rerunfailures': '2.1.0', 'html': '1.16.1', 'base-url': '1.4.1', 'metadata': '1.5.1'}, 'Packages': {'py': '1.4.33', 'pytest': '3.1.3', 'pluggy': '0.4.0'}}
baseurl: https://standby.mdn.mozit.cloud
rootdir: /app, inifile:
plugins: selenium-1.11.4, html-1.16.1, rerunfailures-2.1.0, variables-1.7.1, metadata-1.5.1, base-url-1.4.1
collected 7 items

tests/functional/test_search.py FR......

------------------------------------ generated html file: /results/pytest.html ------------------------------------
==================================================== FAILURES =====================================================
______________________________________________ test_search_homepage _______________________________________________

base_url = 'https://standby.mdn.mozit.cloud'
selenium = <selenium.webdriver.remote.webdriver.WebDriver (session="fb87753e-dc8f-467a-ab6e-9228f569d0e1")>

    @pytest.mark.smoke
    @pytest.mark.search
    @pytest.mark.nondestructive
    def test_search_homepage(base_url, selenium):
        # open homepage
        page = HomePage(selenium, base_url).open()
        # search for CSS in big box
        search = page.search_for_term(SEARCH_TERM)
        # search term is in search box
>       assert search.search_input_value == SEARCH_TERM

tests/functional/test_search.py:23:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/pages/search.py:31: in search_input_value
    return self.find_element(*self._search_input_locator).get_attribute('value')
/usr/local/lib/python2.7/site-packages/pypom/view.py:32: in find_element
    return self.selenium.find_element(strategy, locator)
/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py:955: in find_element
    'value': value})['value']
/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py:312: in execute
    self.error_handler.check_response(response)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <selenium.webdriver.remote.errorhandler.ErrorHandler object at 0x7f1e3f9bfcd0>
response = {'status': 404, 'value': '{"value":{"error":"no such element","message":"Unable to locate element: [id=\"search-q\"]",...Error@chrome://marionette/content/error.js:463:5\nelement.find/</<@chrome://marionette/content/element.js:291:16\n"}}'}

    def check_response(self, response):
        """
            Checks that a JSON response from the WebDriver does not have an error.

            :Args:
             - response - The JSON response from the WebDriver server as a dictionary
               object.

            :Raises: If the response contains an error message.
            """
        status = response.get('status', None)
        if status is None or status == ErrorCode.SUCCESS:
            return
        value = None
        message = response.get("message", "")
        screen = response.get("screen", "")
        stacktrace = None
        if isinstance(status, int):
            value_json = response.get('value', None)
            if value_json and isinstance(value_json, basestring):
                import json
                try:
                    value = json.loads(value_json)
                    if len(value.keys()) == 1:
                        value = value['value']
                    status = value.get('error', None)
                    if status is None:
                        status = value["status"]
                        message = value["value"]
                        if not isinstance(message, basestring):
                            value = message
                            message = message.get('message')
                    else:
                        message = value.get('message', None)
                except ValueError:
                    pass

        exception_class = ErrorInResponseException
        if status in ErrorCode.NO_SUCH_ELEMENT:
            exception_class = NoSuchElementException
        elif status in ErrorCode.NO_SUCH_FRAME:
            exception_class = NoSuchFrameException
        elif status in ErrorCode.NO_SUCH_WINDOW:
            exception_class = NoSuchWindowException
        elif status in ErrorCode.STALE_ELEMENT_REFERENCE:
            exception_class = StaleElementReferenceException
        elif status in ErrorCode.ELEMENT_NOT_VISIBLE:
            exception_class = ElementNotVisibleException
        elif status in ErrorCode.INVALID_ELEMENT_STATE:
            exception_class = InvalidElementStateException
        elif status in ErrorCode.INVALID_SELECTOR \
                or status in ErrorCode.INVALID_XPATH_SELECTOR \
                or status in ErrorCode.INVALID_XPATH_SELECTOR_RETURN_TYPER:
            exception_class = InvalidSelectorException
        elif status in ErrorCode.ELEMENT_IS_NOT_SELECTABLE:
            exception_class = ElementNotSelectableException
        elif status in ErrorCode.ELEMENT_NOT_INTERACTABLE:
            exception_class = ElementNotInteractableException
        elif status in ErrorCode.INVALID_COOKIE_DOMAIN:
            exception_class = InvalidCookieDomainException
        elif status in ErrorCode.UNABLE_TO_SET_COOKIE:
            exception_class = UnableToSetCookieException
        elif status in ErrorCode.TIMEOUT:
            exception_class = TimeoutException
        elif status in ErrorCode.SCRIPT_TIMEOUT:
            exception_class = TimeoutException
        elif status in ErrorCode.UNKNOWN_ERROR:
            exception_class = WebDriverException
        elif status in ErrorCode.UNEXPECTED_ALERT_OPEN:
            exception_class = UnexpectedAlertPresentException
        elif status in ErrorCode.NO_ALERT_OPEN:
            exception_class = NoAlertPresentException
        elif status in ErrorCode.IME_NOT_AVAILABLE:
            exception_class = ImeNotAvailableException
        elif status in ErrorCode.IME_ENGINE_ACTIVATION_FAILED:
            exception_class = ImeActivationFailedException
        elif status in ErrorCode.MOVE_TARGET_OUT_OF_BOUNDS:
            exception_class = MoveTargetOutOfBoundsException
        elif status in ErrorCode.JAVASCRIPT_ERROR:
            exception_class = JavascriptException
        elif status in ErrorCode.SESSION_NOT_CREATED:
            exception_class = SessionNotCreatedException
        elif status in ErrorCode.INVALID_ARGUMENT:
            exception_class = InvalidArgumentException
        elif status in ErrorCode.NO_SUCH_COOKIE:
            exception_class = NoSuchCookieException
        elif status in ErrorCode.UNABLE_TO_CAPTURE_SCREEN:
            exception_class = ScreenshotException
        elif status in ErrorCode.ELEMENT_CLICK_INTERCEPTED:
            exception_class = ElementClickInterceptedException
        elif status in ErrorCode.INSECURE_CERTIFICATE:
            exception_class = InsecureCertificateException
        elif status in ErrorCode.INVALID_COORDINATES:
            exception_class = InvalidCoordinatesException
        elif status in ErrorCode.INVALID_SESSION_ID:
            exception_class = InvalidSessionIdException
        elif status in ErrorCode.UNKNOWN_METHOD:
            exception_class = UnknownMethodException
        else:
            exception_class = WebDriverException
        if value == '' or value is None:
            value = response['value']
        if isinstance(value, basestring):
            if exception_class == ErrorInResponseException:
                raise exception_class(response, value)
            raise exception_class(value)
        if message == "" and 'message' in value:
            message = value['message']

        screen = None
        if 'screen' in value:
            screen = value['screen']

        stacktrace = None
        if 'stackTrace' in value and value['stackTrace']:
            stacktrace = []
            try:
                for frame in value['stackTrace']:
                    line = self._value_or_default(frame, 'lineNumber', '')
                    file = self._value_or_default(frame, 'fileName', '<anonymous>')
                    if line:
                        file = "%s:%s" % (file, line)
                    meth = self._value_or_default(frame, 'methodName', '<anonymous>')
                    if 'className' in frame:
                        meth = "%s.%s" % (frame['className'], meth)
                    msg = "    at %s (%s)"
                    msg = msg % (meth, file)
                    stacktrace.append(msg)
            except TypeError:
                pass
        if exception_class == ErrorInResponseException:
            raise exception_class(response, message)
        elif exception_class == UnexpectedAlertPresentException and 'alert' in value:
            raise exception_class(message, screen, stacktrace, value['alert'].get('text'))
>       raise exception_class(message, screen, stacktrace)
E       NoSuchElementException: Message: Unable to locate element: [id="search-q"]

/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py:237: NoSuchElementException
------------------------------------------------- pytest-selenium -------------------------------------------------
URL: https://standby.mdn.mozit.cloud/en-US/search?q=css&topic=apps&topic=html&topic=css&topic=js&topic=api&topic=canvas&topic=svg&topic=webgl&topic=mobile&topic=webdev&topic=http&topic=webext&topic=standards
================================== 1 failed, 6 passed, 1 rerun in 57.14 seconds ===================================
*** Shutting down dockerized browsers...
selenium-firefox-kuma
selenium-firefox-kuma
*** Test results in test_results/functional_20181003_1627

escattone commented 6 years ago

Headless Tests

We will exclude the tests within tests/headless/tes_cdn.py, since those depend on the use of a CDN.

Headless tests require a Python environment with the requirements in requirements/test.txt. See Setting up test virtual environment. Run the headless tests, which make safe HTTP requests (no POSTs, no logins):

pytest --maintenance-mode --base-url https://standby.mdn.mozit.cloud tests/headless/test_endpoints.py tests/headless/test_redirects.py tests/headless/test_robots.py

[x] Headless tests -- except those within tests/headless/tes_cdn.py -- pass

(mdntest) rjohnson-25186:kuma rjohnson$ pytest --maintenance-mode --base-url https://standby.mdn.mozit.cloud tests/headless/test_endpoints.py tests/headless/test_redirects.py tests/headless/test_robots.py
=============================================== test session starts ===============================================
platform darwin -- Python 2.7.12, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
sensitiveurl: .* *** WARNING: sensitive url matches https://standby.mdn.mozit.cloud ***
metadata: {'Python': '2.7.12', 'Driver': None, 'Capabilities': {}, 'Base URL': 'https://standby.mdn.mozit.cloud', 'Platform': 'Darwin-17.7.0-x86_64-i386-64bit', 'kuma': {u'services': {u'search': {u'available': True, u'count': 60179, u'populated': True}, u'kumascript': {u'available': True, u'revision': u'725538d72c9713589a89fca9f8c771da59c396a0'}, u'test_accounts': {u'available': False}, u'database': {u'available': True, u'document_count': 130020, u'populated': True}}, u'version': 1, u'request': {u'url': u'https://standby.mdn.mozit.cloud/_kuma_status.json', u'is_secure': True, u'host': u'standby.mdn.mozit.cloud', u'scheme': u'https'}, 'response': {'headers': {'Content-Length': '1294', 'strict-transport-security': 'max-age=63072000', 'x-content-type-options': 'nosniff', 'Content-Language': 'en-US', 'Expires': 'Wed, 03 Oct 2018 23:55:34 GMT', 'Server': 'meinheld/0.6.1', 'Connection': 'keep-alive', 'x-xss-protection': '1; mode=block', 'Cache-Control': 'no-cache, no-store, must-revalidate, max-age=0', 'Date': 'Wed, 03 Oct 2018 23:55:34 GMT', 'X-Frame-Options': 'DENY', 'Content-Type': 'application/json'}}, u'settings': {u'ATTACHMENT_HOST': u'mdn.mozillademos.org', u'PROTOCOL': u'https://', u'INTERACTIVE_EXAMPLES_BASE': u'https://interactive-examples.mdn.mozilla.net', u'MAINTENANCE_MODE': True, u'STATIC_URL': u'https://developer.mozilla.org/static/', u'SITE_URL': u'https://developer.mozilla.org', u'ATTACHMENT_ORIGIN': u'prod.mdn.mozit.cloud', u'DEBUG': False, u'ALLOWED_HOSTS': [u'developer.mozilla.org', u'cdn.mdn.mozilla.net', u'mdn.mozillademos.org', u'demos.mdn.mozit.cloud', u'developer-prod.mdn.mozit.cloud', u'prod.mdn.mozit.cloud', u'standby.mdn.mozit.cloud', u'developer.mozilla.com', u'mdn.mozilla.org', u'developer-new.mozilla.org', u'developers.mozilla.org'], u'REVISION_HASH': u'135eceb4e68afa16125fbf5bff77349925a3c87d', u'LEGACY_HOSTS': [u'cdn.mdn.mozilla.net', u'developer.mozilla.com', u'mdn.mozilla.org', u'developer-new.mozilla.org', u'developers.mozilla.org']}}, 'Plugins': {'variables': '1.7.1', 'selenium': '1.11.4', 'xdist': '1.16.0', 'rerunfailures': '2.1.0', 'html': '1.16.1', 'base-url': '1.4.1', 'metadata': '1.5.1'}, 'Packages': {'py': '1.5.3', 'pytest': '3.5.1', 'pluggy': '0.6.0'}}
baseurl: https://standby.mdn.mozit.cloud
rootdir: /Users/rjohnson/repos/kuma, inifile: pytest.ini
plugins: xdist-1.16.0, variables-1.7.1, selenium-1.11.4, rerunfailures-2.1.0, metadata-1.5.1, html-1.16.1, base-url-1.4.1
collected 1295 items

tests/headless/test_endpoints.py ...                                                                        [  0%]
tests/headless/test_redirects.py ...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... [  0%]
tests/headless/test_robots.py ......

========================================== 1295 passed in 977.22 seconds ==========================================

escattone commented 6 years ago

Full Manual Tests

Content tests

[x] https://standby.mdn.mozit.cloud/@api/deki/files/3613/=hut.jpg - 200, image of a hat
[x] https://standby.mdn.mozit.cloud/contribute.json - 200, project info
[x] https://standby.mdn.mozit.cloud/diagrams/workflow/workflow.svg - 200, SVG with images
[x] https://standby.mdn.mozit.cloud/en-US/Firefox/Releases - 200, Firefox logo, list of releases (zoned URL)
[x] https://standby.mdn.mozit.cloud/en-US/dashboards/macros - 200, list of macros and page counts
[x] https://standby.mdn.mozit.cloud/en-US/dashboards/revisions - 200, list of recent changes
[x] https://standby.mdn.mozit.cloud/en-US/docs/Learn/CSS/Styling_text/Fundamentals#Color - 200, with sample as iframe
[x] https://standby.mdn.mozit.cloud/en-US/docs/Learn/CSS/Styling_text/Fundamentals$toc - 200, HTML table of contents
[x] https://standby.mdn.mozit.cloud/en-US/docs/Web/HTML$children - 200, JSON list of child pages
[x] https://standby.mdn.mozit.cloud/en-US/docs/Web/HTML$compare?locale=en-US&to=1299417&from=1293895 - 200, compares revisions
[x] https://standby.mdn.mozit.cloud/en-US/docs/Web/HTML$history - 200, list of revisions
[x] https://standby.mdn.mozit.cloud/en-US/docs/Web/HTML$json - 200, JSON of page metadata
[x] https://standby.mdn.mozit.cloud/en-US/docs/Web/HTML$revision/1293895 - 200, historical revision
[x] https://standby.mdn.mozit.cloud/en-US/docs/all - 200, paginated list of docs
[x] https://standby.mdn.mozit.cloud/en-US/docs/ckeditor_config.js - 200, JavaScript
[x] https://standby.mdn.mozit.cloud/en-US/docs/feeds/atom/files/ - 200, Atom feed of changed files
[x] https://standby.mdn.mozit.cloud/en-US/docs/feeds/rss/all/ - 200, RSS feed of new pages
[x] https://standby.mdn.mozit.cloud/en-US/docs/feeds/rss/needs-review/ - 200, RSS feed of pages needing review
[x] https://standby.mdn.mozit.cloud/en-US/docs/feeds/rss/needs-review/technical - 200, RSS feed of pages needing technical review
[x] https://standby.mdn.mozit.cloud/en-US/docs/feeds/rss/revisions - 200, RSS feed of changes
[x] https://standby.mdn.mozit.cloud/en-US/docs/feeds/rss/tag/CSS - 200, RSS feed of pages with CSS tag
[x] https://standby.mdn.mozit.cloud/en-US/docs/needs-review/editorial - 200, paginated list of documents
[x] https://standby.mdn.mozit.cloud/en-US/docs/tag/ARIA - 200, list of documents
[x] https://standby.mdn.mozit.cloud/en-US/docs/tags - 200, paginated list of tags
[x] https://standby.mdn.mozit.cloud/en-US/docs/top-level - 200, paginated list of documents
[x] https://standby.mdn.mozit.cloud/en-US/docs/with-errors - 200, (empty?) paginated list of documents
[x] https://standby.mdn.mozit.cloud/en-US/docs/without-parent - 200, paginated list of documents
[x] https://standby.mdn.mozit.cloud/en-US/profiles/sheppy - 200, Sheppy's profile
[x] https://standby.mdn.mozit.cloud/en-US/promote/ - 200, Promote MDN with 4 buttons
[x] https://standby.mdn.mozit.cloud/en-US/search - 200, Search results
[x] https://standby.mdn.mozit.cloud/fellowship/ - 301 to archive of 2015 MDN Fellowship Program
[x] https://standby.mdn.mozit.cloud/files/12984/web-font-example.png - Redirects to https://mdn.mozillademos.org
[x] https://standby.mdn.mozit.cloud/fr/docs/feeds/rss/l10n-updates/ - 200, RSS feed of out-of-date pages
[x] https://standby.mdn.mozit.cloud/fr/docs/localization-tag/inprogress - 200, paginated list of documents
[x] https://standby.mdn.mozit.cloud/humans.txt - 200, list of GitHub usernames
[x] https://standby.mdn.mozit.cloud/media/kumascript-revision.txt - 200, git commit hash for kumascript
[x] https://standby.mdn.mozit.cloud/media/revision.txt - 200, git commit hash for kuma
[x] https://standby.mdn.mozit.cloud/miel - 500 Internal Server Error
[x] https://standby.mdn.mozit.cloud/presentations/microsummaries/index.html - 200, 2006 OSCON presentation
[x] https://standby.mdn.mozit.cloud/robots.txt - 200, robots disallow list
[x] https://standby.mdn.mozit.cloud/samples/webgl/sample3 - 200, Shows WebGL demo
[x] https://standby.mdn.mozit.cloud/sitemap.xml - 200, list of sitemaps
[x] https://standby.mdn.mozit.cloud/sitemaps/en-US/sitemap.xml - 200, list of en-US pages

Anonymous tests

Test these URLs as an anonymous user:

[x] https://standby.mdn.mozit.cloud/admin/users/user/1/ - 302 redirect to the maintenance-mode page
[x] https://standby.mdn.mozit.cloud/en-US/dashboards/spam - 302 redirect to the maintenance-mode page
[x] https://standby.mdn.mozit.cloud/en-US/docs/Web/HTML$edit - 302 redirect to the maintenance-mode page
[x] https://standby.mdn.mozit.cloud/en-US/docs/Web/HTML$translate - 302 redirect to the maintenance-mode page
[x] https://standby.mdn.mozit.cloud/en-US/Firefox/Releases$edit - 302 redirect to the maintenance-mode page
[x] https://standby.mdn.mozit.cloud/en-US/Firefox/Releases$translate - 302 redirect to the maintenance-mode page

escattone commented 6 years ago

This is done. The standby instance is running in maintenance-mode within the MozIT standby cluster and passes all tests.

mdn / infra

Setup Standby (Disaster-Recovery) instance of MDN #78

User story

Acceptance Criteria

Tasks:

The standby instance

Manual Sanity Check

Home page

Article page

Functional Tests

Headless Tests

Full Manual Tests

Content tests

Anonymous tests