18F / development-guide

A set of guidelines and best practices for an awesome software engineering team
https://engineering.18f.gov
Other
1.25k stars 122 forks source link

Add recommendations around reducing big bang release risk #257

Closed Sgtpluck closed 3 years ago

Sgtpluck commented 3 years ago

Addressing #230 and #232

github-actions[bot] commented 3 years ago
Pa11y testing results ``` > development-guide@1.0.0 pa11y-ci:sitemap /home/runner/work/development-guide/development-guide > pa11y-ci --sitemap https://engineering.18f.gov/sitemap.xml --sitemap-find https://engineering.18f.gov --sitemap-replace http://localhost:4000 --sitemap-exclude "/*.pdf" Running Pa11y on 38 URLs: > http://localhost:4000/architecture-reviews/ - 0 errors > http://localhost:4000/architecture-reviews/data-act-pilot/ - 0 errors > http://localhost:4000/architecture-reviews/micro-purchase/ - 0 errors > http://localhost:4000/accessibility-scanning/ - 0 errors > http://localhost:4000/browser-testing/ - 0 errors > http://localhost:4000/code-review/ - 0 errors > http://localhost:4000/continuous-deployment/ - 0 errors > http://localhost:4000/datastore-selection/ - 0 errors > http://localhost:4000/development-environments/ - 0 errors > http://localhost:4000/css/ - 0 errors > http://localhost:4000/docker/ - 0 errors > http://localhost:4000/frontend/ - 0 errors > http://localhost:4000/example-workflows/ - 0 errors > http://localhost:4000/incident-reports/ - 0 errors > http://localhost:4000/incident-reports/cloud-gov/ - 0 errors > http://localhost:4000/ - 0 errors > http://localhost:4000/language-selection/ - 0 errors > http://localhost:4000/integrations/ - 0 errors > http://localhost:4000/javascript/ - 0 errors > http://localhost:4000/laptop-setup/ - 0 errors > http://localhost:4000/markdown/ - 0 errors > http://localhost:4000/license/ - 0 errors > http://localhost:4000/nodejs/ - 0 errors > http://localhost:4000/people/ - 0 errors > http://localhost:4000/people/assessment/ - 0 errors > http://localhost:4000/resources/ - 0 errors > http://localhost:4000/project-setup/ - 0 errors > http://localhost:4000/python/ - 0 errors > http://localhost:4000/ruby/ - 0 errors > http://localhost:4000/security/ - 0 errors > http://localhost:4000/security/cloud-services/ - 0 errors > http://localhost:4000/security/content-security-policy/ - 0 errors > http://localhost:4000/security/dependency-remediation/ - 0 errors > http://localhost:4000/security/output-encoding/ - 0 errors > http://localhost:4000/security/incident-response-drills/ - 0 errors > http://localhost:4000/sharepoint/ - 0 errors > http://localhost:4000/workflow/ - 0 errors > http://localhost:4000/web-architecture/ - 0 errors ✔ 38/38 URLs passed ```
github-actions[bot] commented 3 years ago
Pa11y testing results ``` > development-guide@1.0.0 pa11y-ci:sitemap /home/runner/work/development-guide/development-guide > pa11y-ci --sitemap https://engineering.18f.gov/sitemap.xml --sitemap-find https://engineering.18f.gov --sitemap-replace http://localhost:4000 --sitemap-exclude "/*.pdf" Running Pa11y on 38 URLs: > http://localhost:4000/architecture-reviews/ - 0 errors > http://localhost:4000/architecture-reviews/data-act-pilot/ - 0 errors > http://localhost:4000/architecture-reviews/micro-purchase/ - 0 errors (node:1909) UnhandledPromiseRejectionWarning: Error: Page crashed! at Page._onTargetCrashed (/home/runner/work/development-guide/development-guide/node_modules/puppeteer/lib/Page.js:215:24) at CDPSession. (/home/runner/work/development-guide/development-guide/node_modules/puppeteer/lib/Page.js:123:56) at CDPSession.emit (events.js:376:20) at CDPSession._onMessage (/home/runner/work/development-guide/development-guide/node_modules/puppeteer/lib/Connection.js:200:12) at Connection._onMessage (/home/runner/work/development-guide/development-guide/node_modules/puppeteer/lib/Connection.js:112:17) at WebSocket. (/home/runner/work/development-guide/development-guide/node_modules/puppeteer/lib/WebSocketTransport.js:44:24) at WebSocket.onMessage (/home/runner/work/development-guide/development-guide/node_modules/ws/lib/event-target.js:120:16) at WebSocket.emit (events.js:376:20) at Receiver.receiverOnMessage (/home/runner/work/development-guide/development-guide/node_modules/ws/lib/websocket.js:789:20) at Receiver.emit (events.js:376:20) (Use `node --trace-warnings ...` to show where the warning was created) (node:1909) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1) (node:1909) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code. > http://localhost:4000/accessibility-scanning/ - 0 errors > http://localhost:4000/continuous-deployment/ - 0 errors > http://localhost:4000/code-review/ - 0 errors > http://localhost:4000/datastore-selection/ - 0 errors > http://localhost:4000/development-environments/ - 0 errors > http://localhost:4000/css/ - 0 errors > http://localhost:4000/docker/ - 0 errors > http://localhost:4000/example-workflows/ - 0 errors > http://localhost:4000/frontend/ - 0 errors > http://localhost:4000/incident-reports/ - 0 errors > http://localhost:4000/incident-reports/cloud-gov/ - 0 errors > http://localhost:4000/ - 0 errors > http://localhost:4000/integrations/ - 0 errors > http://localhost:4000/language-selection/ - 0 errors > http://localhost:4000/javascript/ - 0 errors > http://localhost:4000/laptop-setup/ - 0 errors > http://localhost:4000/license/ - 0 errors > http://localhost:4000/markdown/ - 0 errors > http://localhost:4000/nodejs/ - 0 errors > http://localhost:4000/people/ - 0 errors > http://localhost:4000/people/assessment/ - 0 errors > http://localhost:4000/project-setup/ - 0 errors > http://localhost:4000/python/ - 0 errors > http://localhost:4000/resources/ - 0 errors > http://localhost:4000/ruby/ - 0 errors > http://localhost:4000/security/ - 0 errors > http://localhost:4000/security/cloud-services/ - 0 errors > http://localhost:4000/security/content-security-policy/ - 0 errors > http://localhost:4000/security/dependency-remediation/ - 0 errors > http://localhost:4000/security/incident-response-drills/ - 0 errors > http://localhost:4000/security/output-encoding/ - 0 errors > http://localhost:4000/sharepoint/ - 0 errors > http://localhost:4000/workflow/ - 0 errors > http://localhost:4000/web-architecture/ - 0 errors > http://localhost:4000/browser-testing/ - Failed to run Errors in http://localhost:4000/browser-testing/: • Error: Pa11y timed out (30000ms) ✘ 37/38 URLs passed npm ERR! code ELIFECYCLE npm ERR! errno 2 npm ERR! development-guide@1.0.0 pa11y-ci:sitemap: `pa11y-ci --sitemap https://engineering.18f.gov/sitemap.xml --sitemap-find https://engineering.18f.gov --sitemap-replace http://localhost:4000 --sitemap-exclude "/*.pdf"` npm ERR! Exit status 2 npm ERR! npm ERR! Failed at the development-guide@1.0.0 pa11y-ci:sitemap script. npm ERR! This is probably not a problem with npm. There is likely additional logging output above. npm ERR! A complete log of this run can be found in: npm ERR! /home/runner/.npm/_logs/2021-05-21T13_31_25_547Z-debug.log ```
github-actions[bot] commented 3 years ago
Pa11y testing results ``` > development-guide@1.0.0 pa11y-ci:sitemap /home/runner/work/development-guide/development-guide > pa11y-ci --sitemap https://engineering.18f.gov/sitemap.xml --sitemap-find https://engineering.18f.gov --sitemap-replace http://localhost:4000 --sitemap-exclude "/*.pdf" Running Pa11y on 38 URLs: > http://localhost:4000/architecture-reviews/ - 0 errors > http://localhost:4000/architecture-reviews/data-act-pilot/ - 0 errors > http://localhost:4000/architecture-reviews/micro-purchase/ - 0 errors > http://localhost:4000/accessibility-scanning/ - 0 errors > http://localhost:4000/browser-testing/ - 0 errors > http://localhost:4000/code-review/ - 0 errors > http://localhost:4000/continuous-deployment/ - 0 errors > http://localhost:4000/datastore-selection/ - 0 errors > http://localhost:4000/development-environments/ - 0 errors > http://localhost:4000/docker/ - 0 errors > http://localhost:4000/example-workflows/ - 0 errors > http://localhost:4000/css/ - 0 errors > http://localhost:4000/frontend/ - 0 errors > http://localhost:4000/incident-reports/cloud-gov/ - 0 errors > http://localhost:4000/incident-reports/ - 0 errors > http://localhost:4000/ - 0 errors > http://localhost:4000/integrations/ - 0 errors > http://localhost:4000/language-selection/ - 0 errors > http://localhost:4000/laptop-setup/ - 0 errors > http://localhost:4000/javascript/ - 0 errors > http://localhost:4000/license/ - 0 errors > http://localhost:4000/markdown/ - 0 errors > http://localhost:4000/nodejs/ - 0 errors > http://localhost:4000/people/ - 0 errors > http://localhost:4000/people/assessment/ - 0 errors > http://localhost:4000/project-setup/ - 0 errors > http://localhost:4000/resources/ - 0 errors > http://localhost:4000/ruby/ - 0 errors > http://localhost:4000/security/ - 0 errors > http://localhost:4000/python/ - 0 errors > http://localhost:4000/security/cloud-services/ - 0 errors > http://localhost:4000/security/content-security-policy/ - 0 errors > http://localhost:4000/security/dependency-remediation/ - 0 errors > http://localhost:4000/security/incident-response-drills/ - 0 errors > http://localhost:4000/security/output-encoding/ - 0 errors > http://localhost:4000/sharepoint/ - 0 errors > http://localhost:4000/web-architecture/ - 0 errors > http://localhost:4000/workflow/ - 0 errors ✔ 38/38 URLs passed ```
perldude commented 3 years ago

Is there any interest in having a rollback plan should the big release catastrophically fail? Sure, beta testing should keep that from happening but Murphy is always a factor.

-Michael Sullivan

tbaxter-18f commented 3 years ago

I think @Sgtpluck mentioned that in another PR that picked up after this (it got complicated)