Contract Testing - Release Pact to enable testing of integrations on VA.gov

alexpappasoddball commented 4 years ago

NOTE: the name of this initiative has been changed from Integrated e2e Testing to Contract Testing to reflect the chosen approach. (MK 9/15/20)

Problem Statement

The current process to test a frontend application (vets-website) with the backend (vets-api) is not suitable for the long term needs of the platform. The current process involves using a combination of Swagger docs, betamocks, JSON Schema and VCR recordings. These tools are theoretical tools and don’t actually reflect the current state of the APIs, which could lead them to being out of sync with what the actual response is expecting. An additional pain point is that in order for an application team to test their app with these backend tools they have to run both applications at the same time. The current solution is only testable in the staging environment as well.

How might we enable VFS teams to test integrations with vets-api in a non-production environment? How might we enable VSP to more easily confirm that platform-wide changes are non-breaking?

Hypothesis or Bet

We hypothesize that if we provide VFS teams with a supported way to test FE & BE integrations, then fewer breaking changes will be released to production.

We will know we're done when... ("Definition of Done")

[x] Solution for testing integrations identified and agreed upon
[ ] Solution is set up and launched for use by VFS teams
[x] Documentation & other training materials (as applicable) are written
[x] VSP policy around use of the tool is in place

Launch Checklist

Guidance (delete before posting)

Is this service / tool / feature...

... tested?

[x] Usability test (TODO: link) has been performed, to validate that new changes enable users to do what was intended and that these changes don't worsen quality elsewhere. If usability test isn't relevant for this change, document the reason for skipping it.
- [x] ... and issues discovered in usability testing have been addressed.
- Note on skipping: metrics that show the impact of before/after can be a substitute for usability testing.
[x] End-to-end manual QA or UAT is complete, to validate there are no high-severity issues before launching

... documented?

[x] New documentation is written pursuant to our documentation style guide
[x] Product is included in the List of VSP Products
- List the existing product that this initiative fits within, or add a new product to this list.
[x] Internal-facing: there's a Product Outline checked into products/platform/PRODUCT_NAME/
- Note: the Product Directory Name should match 1:1 with the List of VSP Products
[x] External-facing: a VFS-facing README exists for this product/feature tool
- [x] ... and should be located at platform/PRODUCT_NAME/README.md
[x] External-facing: a User Guide exists for this product/feature/tool, and is updated for changes from this initiative
- [x] ... and should be linked from the VFS-facing README for your product
- [x] ... and should be located within platform/PRODUCT_NAME/, unless you already have another location for it
- [ ] (if applicable)... and post to #vsp-content-ia about whether this should be added to the Documentation homepage
[x] (if applicable) Post to #vsp-service-design for external communication about this change (e.g. VSP Newsletter, customer-facing meetings)

... measurable

[x] (if applicable) This change has clearly-defined success metrics, with instrumentation of those analytics where possible, or a reason documented for skipping it.
- For help, see: Analytics team
[x] This change has an accompanying VSP Initiative Release Plan.

Required Artifacts

Documentation

PRODUCT_NAME: directory name used for your product documentation
Product Outline: contract-testing-product-outline.md
README: README.md
User Guide: how-to-use-pact-for-contract-testing.md

Testing

Usability test: with VSA Facilities & with VSA Authenticated Experience teams
Manual QA: link to GitHub issue or documented results
Automated tests: link to tests, or "N/A"

Measurement

Success metrics: KPIs in product outline
Release plan: WIP — drafted before new release plan ticket https://github.com/department-of-veterans-affairs/va.gov-team/blob/master/products/platform/contract-testing/pact-release-plan.md

TODOs

[x] Convert this issue to an epic
[x] Add your team's label to this epic

LindseySaari commented 4 years ago

[x] HCA example up and running & broker running locally
[x] Building out search

LindseySaari commented 4 years ago

It was discovered that there’s some limitations in our mock data related to search. The frontend piece defines three contract specifications (multiple, single and no result cases). We currently only have mock data for the multiple results example (default.yml in vets-api-mockdata). I created new fe and be branches to test this all out/allow search contracts to verify and pass. The front-end specifications requires the mock data to contain specific fields and the mock data doesn’t align. The search example doesn’t require a provider state because the call to the search endpoint is mocked. The next step may be testing out a more complex endpoint that calls out to a different external service or authorization and figure out how to integrate with CircleCI.

Frontend Branch Backend Branch Search uses this mock data (had to tweak a bit to get contracts to verify)

Instructions on how to verify and publish the PACTs FE Search Draft PR example

meganhkelley commented 4 years ago

Previous title and epic contents: Integrated e2e Testing — Develop an integration testing solution to ensure FE and BE are tested with e2e testing

Problem

The current process to test a frontend application (vets-website) with the backend (vets-api) is not suitable for the long term needs of the platform. The current process involves using a combination of Swagger docs, betamocks, JSON Schema and VCR recordings. These tools are theoretical tools and don't actually reflect the current state of the apis which could lead them to being out of sync with what the actual response is expecting. In additional pain point is in order for an application team to test their app with these backend tools they have to run both applications at the same time. The current solution is only testable in the staging environment as well.

Goal

Allow an accurate, easy to use process for application developers to test frontend applications against the backend.

Details and Notes

There have been previous trials within the VSP to explore using Consumer Driven Contract Testing (CDCT) which would allow frontend applications to test their integrations with the backend without having to run both applications simultaneously. Additionally, it would provide a real time (always updated) process to ensure that the requests coming from the frontend will consume the expected responses from the backend.

There are some outstanding questions about what this would look like for down stream services. That could be an additional effort to migrate those services to a CDCT approach.

Objective

Objective 1: Any volume of simultaneous customers have a delightful, self-service, end-to-end experience using the Platform to build on VA.gov.

QASP

VSP can begin to track - Number of top ten VA.gov user experiences, based on unique transactions per month, that have a corresponding automated end-to-end browser test that exercises multiple pieces of functionality including both FE and BE.

Success Metrics

[ ] A standardized approach has been agreed upon with the BE/FE Tools team
[ ] New process is documented clearly and shared
[ ] Application teams can test their API calls without running both applications at the same time
[ ] Responses from the backend are accurate and always up to date.

Measuring Success

TBD based on approach

We will know we're done when

FE + BE agree on an integration testing framework

holdenhinkle commented 3 years ago

Created doc to track baseline KPI values - https://docs.google.com/spreadsheets/d/1W1a5nRlAqdS4Itf13r6bUmSHBldgVkAnuqHa2ooh55c/edit#gid=0

These values should be added to the Contract Testing Product Outline - https://github.com/department-of-veterans-affairs/va.gov-team/blob/master/products/platform/contract-testing/contract-testing-product-outline.md#key-performance-indicators-kpis

holdenhinkle commented 3 years ago

Pact baseline KPI research--Postmortems that mention vets-api: https://docs.google.com/document/d/17VbnGu5Tt6wTQW-q_ZZ8AM1QJyjoA-yLjKWlSP62cPY/edit

I made note of postmortems where I think Pact tests could have helped, if they existed. I could be wrong, but if I understand the issues correctly, Pact could have detected the issues.

holdenhinkle commented 3 years ago

Quick Survey of Pact Sentiment on Slack. I couldn't find much but made a list of interesting Pact Posts (including some video demos) along the away - https://docs.google.com/document/d/1VhgwAfzUSW0-9mgV76p-0VO5D51OO8i6IsY471fnNm0/edit#

holdenhinkle commented 3 years ago

PR to update baseline KPI values - https://github.com/department-of-veterans-affairs/va.gov-team/pull/21973

department-of-veterans-affairs / va.gov-team