kpethtel
commented
1 year ago Overview
I don't think contract testing as it's defined here will achieve the goals of this ticket. I believe what's being proposed is closer to schema testing.
One very important caveat to all of this is that while most vets-api client fetch methods return the data unaltered, either as a hash or struct, others process the data into either a Response object or as instances of models. In cases where we receive altered data, there's no way of getting the unaltered response without changing the client code, which isn't a viable option. The system simply isn't designed for it. But for the cases where we receive unaltered data, we can create a small library that can be used to check the schema. See list at bottom for examples of some endpoints where this simply won't work.
Another important question is whether it's worth testing the data received or only the schema. Testing the actual data would be fragile because some data is expected to change over time, so in my opinion it's probably not worth testing.
Options
1) We can create a cron job that periodically runs through a series of endpoint tests. We could keep the expected response schema information in config files. Each new data type would require for code to be added to that uses the appropriate vets-api client to make the call. That will make this relatively difficult to maintain.
Pros:
- allows for testing both schema and body
- does not require any changes to production application code
- minimal performance impact because it would all be done in a cron job
Cons:
- each new test would require writing code for emulating how the controller fetches the data
- would only work with specified test users
- requires maintaining more code
- can drift from the code that's used by our controllers
2) Creating a schema discovery endpoint that only functions on staging. It could be something like mobile/discovery/:data_type?method=get. Similar to option 1, each endpoint would require some code setup to utilize the appropriate vets-api client, which means this will also be difficult to maintain. This endpoint could then be utilized by the staging tester, which would contain all of the config files for the expected responses.
Pros:
- allows for testing both schema and body
- does not require any changes to production application code
- allows us to use the staging tester to make the actual comparisons. The staging tester is designed to make these types of comparisons and it's tempting to find a way to offload the comparisons to it.
Cons:
- each new test would require writing code for emulating how the controller fetches the data
- would only work with specified test users
- requires maintaining more code
- can drift from the code that's used by our controllers
- complex and has multiple points of failure
- requires creating an endpoint for exposing data, which may not be ideal from a data security POV. This should be fine if we limit it to staging.
3) Create a library that can be used within mobile vets-api code.
Pros:
- adding new test cases only requires writing the controller to use the test library and adding config data that defines the expected schema; no need to maintain separate code path that can drift.
- isn't specific to individual test users and can be used in production against live code, which could reveal differences between staging and production
- easily feature flagged for safety
Cons:
- works well for testing schema but isn't a good fit for testing data, unless we design it to only work for specific test users.
- requires changing live code rather than creating a separate test suite
- checking schema on every request seems unnecessary and we may need to limit how often we make the comparison in order to minimize performance impact
Conclusion
I believe that option 3 is our best choice. It has the flexibility to also work on all requests in production, which would give us the widest array of test cases, or to be limited to specific test users in staging.
Of the other two options, I would prefer option 1 because it's less complex.
List of show and indexes where we receive altered data:
- community care providers
- decision letters uses VBMS gem. would need to look into gem for to see how it's handled, but it seems very unlikely to return a model.
- demographics
- EVSS (legacy) disability rating
- folders
- letter beneficiary contains some slight modifications but should still largely work
- EVSS letters and letter beneficiaries have some modifications
- facility data within location show
- messages index and show
- military history
- payment history uses BGS gem and probably modifies data
- payment information
- threads
- triage teams
- message threads
timwright12
Description
On the mobile team we have a perceived issue with upstream services that frequently change and affect the downstream app. The team has to react to these issues as they come up and it often times can disrupt a day's/sprint's work.
This is ticket is to do some discovery work on the possibility of setting up contract tests between the mobile API and upstream services in order to proactively detect schema and data shape changes rather than waiting to logs to alert us.
I've linked to Pact below, which is a contract testing framework that was used with Web platform at one point, but we shouldn't feel tied to it just because it's familiar.
Resources
Definition of Done