Prevent the use of Deploying Color for real Users

[mmarcotte]

As a Systems Administrator, so that I can ensure humans don't inadvertently perform work on the passive/deploying color, I need their access to be restricted. As a Systems Administrator, so that I can ensure test users can perform smoketests on the passive/deploying color, I need their access to be available.

We've recently found that it's possible for a User's browser to become stale and have the wrong color loaded. When they click login, they call the wrong color's login API and if authenticated can begin performing work on the wrong/inactive/passive color. This becomes especially problematic during a migration deployment. It's rare that this would happen in production, but we need to make this impossible.

Pre-Conditions

Acceptance Criteria

• During a deployment, "human users" (non designated "test users") cannot login to the passive/deploying color.
• During a deployment, "test users" (designated "test users") can login to the passive/deploying color and perform automated tests as needed

Notes

• During a deployment both the active and passive colors are available for API calls

Tasks

Test Cases

Story Definition of Ready (updated on 12/23/22)

The following criteria must be met in order for the user story to be picked up by the Flexion development team. The user story must:

• [ ] Is framed in business/user need, the value has been addressed.
• [ ] Includes acceptance criteria
• [ ] Has been refined
• [ ] Pre conditions have been satisfied.

Process: Flexion developers and designers will test if the story meets acceptance criteria and test cases in Flexion dev and staging environments (“standard testing”). If additional acceptance criteria or testing scenarios are discovered while the story is in progress, a new story should be created, added to the backlog and prioritized by the product owner.

Definition of Done (Updated 5-19-22)

Product Owner

• [ ] Acceptance criteria have been met and validated on the Court's migration environment
• [ ] Add scenario to testing document, if applicable (https://docs.google.com/spreadsheets/d/1FUHKC_YrT-PosaWD5gRVmsDzI1HS_U-8CyMIb-qX9EA/edit?usp=sharing)

UX

• [ ] Business test scenarios have been refined to meet all acceptance criteria
• [ ] Usability has been validated
• [ ] Wiki has been updated (if applicable)
• [ ] Story has been tested on a mobile device (for external users only)

Engineering

• [ ] Automated test scripts have been written, including visual tests for newly added PDFs.
• [ ] Field level and page level validation errors (front-end and server-side) integrated and functioning.
• [ ] Verify that language for docket record for internal users and external users is identical.
• [ ] New screens have been added to pa11y scripts.
• [ ] All new functionality verified to work with keyboard and macOS voiceover https://www.apple.com/voiceover/info/guide/_1124.html.
• [ ] READMEs, other appropriate docs, and swagger/APIs fully updated.
• [ ] UI should be touch optimized and responsive for external only (functions on supported mobile devices and optimized for screen sizes as required).
• [ ] Interactors should validate entities before calling persistence methods.
• [ ] Code refactored for clarity and to remove any known technical debt.
• [ ] If new docket entries have been added as seed data to efcms-local.json, 3 local s3 files corresponding to that docketEntryId have been added to web-api/storage/fixtures/s3/noop-documents-local-us-east-1
• [ ] Acceptance criteria for the story has been met.
• [ ] If there are special instructions in order to deploy into the next environment, add them as a comment in the story.
• [ ] If the work completed for the story requires a reindex without a migration, or any other special deploy steps, apply these changes to the following flexion branches:
  • [ ] experimental1
  • [ ] experimental2
  • [ ] experimental3
  • [ ] experimental4
  • [ ] experimental5
  • [ ] experimental6
  • [ ] develop
• [ ] Reviewed by UX on a deployed environment.
• [ ] Reviewed by PO on a deployed environment. Can be deployed to the Court's test environment if prod-like data is required. Otherwise deployed to any experimental environment.
• [ ] Deployed to the Court's staging environment.

[katiecissell]

Pre-refinement questions:

• Are human users logged out when this happens?
  • not forcibly
• Both human and test users both say "(non designated "test users"), guessing this isn't supposed to be for both?

[ttlenard]

Test Cases

1) DAWSON users attempt to log in during a deployment, depending on the user type, User experience varies.

• Precondition - Know the current color (blue or green)

• Navigate to the login page (don't log in)

• Trigger a deployment and wait for it to successfully complete

• Trigger a second deployment and while it is happening:

Expected Results:

• [ ] A real user should NOT be able to log in on the deploying color, and instead would be logged into and operate on the current color
• [ ] A test user should be able to log in on the deploying color, and operate on the deploying color
  • DEV QUESTION - is this the only ones that the cypress tests use? or are these any of the example.com test users?

2) DAWSON deployment completes; Real user that was on the formerly current color get's logged out of DAWSON.

• Be sure that you are logged in as a real user
• Second deployment completes

Expected Results:

• [ ] Real user is logged out of DAWSON

3) DAWSON deployment completes; Test user that was on the deploying color continues to be able to work as if no deployment occurred.

• Be sure that you are logged in as the test user
• Second deployment completes

Expected results:

• [ ] Test user is able to continue work in DAWSON and they are not logged out.

[akuny]

Options explored:

• Option 1: Check deployed-date.txt file and force navigation to home route ('/') if contents of that file have changed

  • Pros:
  • Already solved for
  • Low complexity
  • Simple polling logic on frontend
  • Cons: There will always be a small window of time to allow requests to go through to the old color (roughly 30-60 seconds)

• Option 2: Add middleware to lambdas to check the Dynamo deploy table for the current color and then throw errors if the API color does not match the current color

  • Pros: No window of time for requests to go through to old color: requests would just error out
  • Cons: Every request would need to do a table lookup, which may result in poor performance, more load on system, etc.
  • Notes: We could potentially cache the table lookup value inside the lambda but then we'd be hitting the same window issue we come up against in the option above

• Option 3: Similar to option 2 above, but instead set an environment variable on the lambdas when we switch colors so that the lambda would know that the color has changed, and then throw errors

  • Pros: No window of time for requests to go through to old color: requests would just error out
  • Cons:
  • Adds complexity to deployment pipeline
  • More cognitive overhead to be wary of

• Option 4: Use AWS load balancer instead of hitting blue or green lambdas

  • Pros: Centralized place to control what API can be hit by users
  • Cons:
  • It's a new approach
  • Could complicate smoke tests
  • Costs money
  • Have to verify if AWS load balancers can instantaneously switch colors or would there be a lag (brings us back to the situation in Option 1 above)

[En-8]

We ultimately pursued option 3 mentioned previously. With the way we implemented it, however, the behavior doesn't precisely align with the test cases. Here's the expected behavior with how we're currently achieving this:

• "DAWSON User attempts to log in during deployment" - In this scenario, if the user has a stale blue UI and is logging in via that stale UI in a way that would result in their traffic hitting the deploying blue API, instead of immediately logging into the current color, we display the user a modal saying "DAWSON was updated" and prompt them to click a button that reloads their UI so that they have the latest available version.
• This same behavior occurs if a user is working on blue, then a deployment to green completes -- instead of being logged out, they are prompted to reload their page, and if they still have a valid session will remain logged in.
• "Test Users" will see the same behavior, but only if a real person is manually logging in as a test user. Our test suite is still able to perform tests as needed, but it's not related to the use of the test user. We instead pass a header that allows the test traffic to hit the deploying color.
• We also implemented the same behavior with the public UI. This is a change to how we previously ensured public users were using the most recent version. Instead of auto-refreshing for them on a predefined interval, we show them the same modal that internal users will see and prompt the user to refresh their browser.

@ttlenard , let us know if we need to discuss any of this.

[swongCO]

UX feedback The modal looks good. There should be more space above the button. This is what I'm seeing:

It should look more like this:

[En-8]

@swongCO Good catch! We were tripped up since it only looked like that in Test. Believe we've got this resolved now in Test, here's a screenshot of the modal now: