Simplify CDN configurations after beta launch

escattone commented 5 years ago

Description

The Cloudfront primary (non-wiki) CDN configurations for stage and prod can be significantly simplified after the beta launch.

Acceptance criteria

[x] The behaviors in the stage and prod CloudFront distributions have been modified (this will reduce the number of behaviors from 28 to about 12). The behaviors after the launch should be (in order):
- static/js/libs/ckeditor/build/* (configured same as before -- vary on host header and t query parameter)
- static/* (configured same as before -- vary on host header)
- media/* (configured same as before -- vary on host header)
- */docs/* (vary on host and accept-language headers, django_language cookie -- no other cookies, and all query params)
- users/* (pass through)
- */users/* (pass through)
- */profile/* (pass through)
- */profiles/* (pass through)
- admin/* (pass through)
- api/v1/doc/* (configured same as before)
- api/* (pass through)
- Default (vary on host and accept-language headers, django_language cookie -- no other cookies, and all query params)
[x] Integration tests pass against stage and prod
[x] Inform @limed that the terraform in https://github.com/mdn/infra can be updated to reflect the behaviors in the current stage and prod CloudFront distributions

escattone commented 4 years ago

The MDN Primary stage CDN has been updated per the instructions above. I'll run both the headless tests as well as some manual login tests against stage when the CDN distribution has been fully updated.

escattone commented 4 years ago

Stage Test Results

Manual login test

I was able to successfully login/logout to/from stage.

Headless tests

All of the headless tests passed.

(kuma-test) rjohnson-44864:kuma rjohnson$ pytest --base-url https://developer.allizom.org tests/
============================================ test session starts =============================================
platform darwin -- Python 2.7.10, pytest-3.1.3, py-1.4.33, pluggy-0.4.0
metadata: {'Python': '2.7.10', 'Driver': None, 'Capabilities': {}, 'Base URL': 'https://developer.allizom.org', 'Platform': 'Darwin-18.7.0-x86_64-i386-64bit', 'kuma': {u'services': {u'search': {u'available': True, u'count': 64139, u'populated': True}, u'kumascript': {u'available': True, u'revision': u'4c5e1f69d74654457bf0642b139d4bea20fe7ee2'}, u'test_accounts': {u'available': False}, u'database': {u'available': True, u'document_count': 139827, u'populated': True}}, u'version': 1, u'request': {u'url': u'https://developer.allizom.org/_kuma_status.json', u'is_secure': True, u'host': u'developer.allizom.org', u'scheme': u'https'}, 'response': {'headers': {'X-Cache': 'Miss from cloudfront', 'X-Amz-Cf-Pop': 'HIO50-C1', 'strict-transport-security': 'max-age=63072000', 'x-content-type-options': 'nosniff', 'Content-Language': 'en-US', 'Content-Encoding': 'gzip', 'Transfer-Encoding': 'chunked', 'Expires': 'Mon, 21 Oct 2019 21:32:00 GMT', 'Vary': 'Accept-Encoding', 'Server': 'meinheld/0.6.1', 'Content-Security-Policy-Report-Only': "script-src https://developer.allizom.org www.google-analytics.com cdn.speedcurve.com static.codepen.io 'unsafe-inline' https://wiki.developer.allizom.org https://checkout.stripe.com; style-src https://developer.allizom.org 'unsafe-inline' https://wiki.developer.allizom.org; report-uri https://sentry.prod.mozaws.net/api/72/security/?sentry_release=da912db54a7f39f3a8e3413277af6715dc96195b&sentry_environment=oregon%3Astage&sentry_key=25e652a045b642dfaa310e92e800058a; default-src 'none'; frame-src https://mdn.mozillademos.org https://interactive-examples.mdn.mozilla.net https://mdn.github.io https://www.youtube.com https://jsfiddle.net https://rpm.newrelic.com https://test262.report https://files-stage.mdn.mozit.cloud https://checkout.stripe.com; img-src https://developer.allizom.org data: https://i2.wp.com https://*.githubusercontent.com https://www.google-analytics.com https://mdn.mozillademos.org https://wiki.developer.allizom.org https://files-stage.mdn.mozit.cloud https://*.stripe.com; connect-src https://developer.allizom.org https://wiki.developer.allizom.org https://checkout.stripe.com; font-src https://developer.allizom.org", 'Connection': 'keep-alive', 'x-xss-protection': '1; mode=block', 'X-Amz-Cf-Id': 'Ms9GHbp10JL0sNRS-fK4OTZu_sB_8x0FSWStmOTw_VkEL6MlmMfKhw==', 'Cache-Control': 'no-cache, no-store, must-revalidate, max-age=0', 'Date': 'Mon, 21 Oct 2019 21:32:00 GMT', 'X-Frame-Options': 'DENY', 'Content-Type': 'application/json', 'Via': '1.1 eece508272520f70691e4eebdc5a6dea.cloudfront.net (CloudFront)'}}, u'settings': {u'ATTACHMENT_HOST': u'files-stage.mdn.mozit.cloud', u'PROTOCOL': u'https://', u'INTERACTIVE_EXAMPLES_BASE': u'https://interactive-examples.mdn.mozilla.net', u'MAINTENANCE_MODE': False, u'STATIC_URL': u'/static/', u'SITE_URL': u'https://developer.allizom.org', u'ATTACHMENT_ORIGIN': u'files-stage.mdn.mozit.cloud', u'DEBUG': False, u'ALLOWED_HOSTS': [u'developer.allizom.org', u'beta.developer.allizom.org', u'wiki.developer.allizom.org', u'developer-stage.mdn.mozit.cloud', u'stage.mdn.mozit.cloud', u'files-stage.mdn.mozit.cloud'], u'REVISION_HASH': u'da912db54a7f39f3a8e3413277af6715dc96195b', u'WIKI_SITE_URL': u'https://wiki.developer.allizom.org'}}, 'Plugins': {'variables': '1.7.1', 'selenium': '1.14.0', 'rerunfailures': '2.1.0', 'html': '1.16.1', 'base-url': '1.4.1', 'metadata': '1.5.1'}, 'Packages': {'py': '1.4.33', 'pytest': '3.1.3', 'pluggy': '0.4.0'}}
baseurl: https://developer.allizom.org
sensitiveurl: .* *** WARNING: sensitive url matches https://developer.allizom.org ***
rootdir: /Users/rjohnson/repos/kuma, inifile: pytest.ini
plugins: metadata-1.5.1, base-url-1.4.1, variables-1.7.1, html-1.16.1, selenium-1.14.0, rerunfailures-2.1.0
collected 1727 items

tests/headless/test_cdn.py ................................................................
tests/headless/test_endpoints.py .............................................................................................................................................................................................................................................................................................................................................................................
tests/headless/test_redirects.py ............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
tests/headless/test_robots.py ......

======================================= 1727 passed in 192.83 seconds ========================================
(kuma-test) rjohnson-44864:kuma rjohnson$

escattone commented 4 years ago

The MDN Primary prod CDN has been updated per the instructions above. Again, I'll run both the headless tests as well as some manual login tests against prod when the CDN distribution has been fully updated.

escattone commented 4 years ago

Prod Test Results

Manual login test

I was able to successfully login to prod as well as logout.

Headless tests

There was one headless test failure, but it is an expected issue on prod (see https://github.com/mdn/kuma/issues/6022).

(kuma-test) rjohnson-44864:kuma rjohnson$ pytest --base-url https://developer.mozilla.org tests/
============================================ test session starts =============================================
platform darwin -- Python 2.7.10, pytest-3.1.3, py-1.4.33, pluggy-0.4.0
metadata: {'Python': '2.7.10', 'Driver': None, 'Capabilities': {}, 'Base URL': 'https://developer.mozilla.org', 'Platform': 'Darwin-18.7.0-x86_64-i386-64bit', 'kuma': {u'services': {u'search': {u'available': True, u'count': 64557, u'populated': True}, u'kumascript': {u'available': True, u'revision': u'4c5e1f69d74654457bf0642b139d4bea20fe7ee2'}, u'test_accounts': {u'available': False}, u'database': {u'available': True, u'document_count': 140479, u'populated': True}}, u'version': 1, u'request': {u'url': u'https://developer.mozilla.org/_kuma_status.json', u'is_secure': True, u'host': u'developer.mozilla.org', u'scheme': u'https'}, 'response': {'headers': {'X-Cache': 'Miss from cloudfront', 'X-Amz-Cf-Pop': 'HIO50-C1', 'strict-transport-security': 'max-age=63072000', 'x-content-type-options': 'nosniff', 'Content-Language': 'en-US', 'Content-Encoding': 'gzip', 'Transfer-Encoding': 'chunked', 'Expires': 'Mon, 21 Oct 2019 22:04:35 GMT', 'Vary': 'Accept-Encoding', 'Server': 'meinheld/0.6.1', 'Connection': 'keep-alive', 'x-xss-protection': '1; mode=block', 'X-Amz-Cf-Id': '9qkArIJf16ROSAj07i6e_l2dCEkBpt6DQhfhF_Qsd_Kf0q7Pm6-gRw==', 'Cache-Control': 'no-cache, no-store, must-revalidate, max-age=0', 'Date': 'Mon, 21 Oct 2019 22:04:35 GMT', 'X-Frame-Options': 'DENY', 'Content-Type': 'application/json', 'Via': '1.1 dbeb98709410a42541e3a4522fba8a67.cloudfront.net (CloudFront)'}}, u'settings': {u'ATTACHMENT_HOST': u'mdn.mozillademos.org', u'PROTOCOL': u'https://', u'INTERACTIVE_EXAMPLES_BASE': u'https://interactive-examples.mdn.mozilla.net', u'MAINTENANCE_MODE': False, u'STATIC_URL': u'/static/', u'SITE_URL': u'https://developer.mozilla.org', u'ATTACHMENT_ORIGIN': u'demos-origin.mdn.mozit.cloud', u'DEBUG': False, u'ALLOWED_HOSTS': [u'developer.mozilla.org', u'beta.developer.mozilla.org', u'wiki.developer.mozilla.org', u'mdn.mozillademos.org', u'demos.mdn.mozit.cloud', u'demos-origin.mdn.mozit.cloud', u'developer-prod.mdn.mozit.cloud', u'prod.mdn.mozit.cloud'], u'REVISION_HASH': u'da912db54a7f39f3a8e3413277af6715dc96195b', u'WIKI_SITE_URL': u'https://wiki.developer.mozilla.org'}}, 'Plugins': {'variables': '1.7.1', 'selenium': '1.14.0', 'rerunfailures': '2.1.0', 'html': '1.16.1', 'base-url': '1.4.1', 'metadata': '1.5.1'}, 'Packages': {'py': '1.4.33', 'pytest': '3.1.3', 'pluggy': '0.4.0'}}
baseurl: https://developer.mozilla.org
sensitiveurl: .* *** WARNING: sensitive url matches https://developer.mozilla.org ***
rootdir: /Users/rjohnson/repos/kuma, inifile: pytest.ini
plugins: metadata-1.5.1, base-url-1.4.1, variables-1.7.1, html-1.16.1, selenium-1.14.0, rerunfailures-2.1.0
collected 1729 items

tests/headless/test_cdn.py ................................................................
tests/headless/test_endpoints.py .......................................................F.....................................................................................................................................................................................................................................................................................................................
tests/headless/test_redirects.py ............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
tests/headless/test_robots.py ........

================================================== FAILURES ==================================================
_____________________________________________ test_user_document _____________________________________________
Traceback (most recent call last):
  File "/Users/rjohnson/repos/kuma/tests/headless/test_endpoints.py", line 136, in test_user_document
    assert content == 'noindex, nofollow'
AssertionError: assert 'index, follow' == 'noindex, nofollow'
  - index, follow
  + noindex, nofollow
  ? ++       ++
================================== 1 failed, 1728 passed in 167.01 seconds ===================================
(kuma-test) rjohnson-44864:kuma rjohnson$

escattone commented 4 years ago

@limed Both the MDN Primary stage CDN and MDN Primary prod CDN have been updated and are ready to be reflected into your Terraform code.

mdn / infra