adrianhall
commented
4 years ago Notes from Architecture Board:
Recording: https://msit.microsoftstream.com/video/dab58b12-8ebe-49ed-8ace-876af461db84
PR #894 (Draft)
1) 3 discussion points
- Jeffrey notes about recordings and how they work. There are many things we have to ignore - it generally works, but someone has to identify whether a field is valid or needs to be ignored.
- How to deal with situations where the service team breaks the endpoint APIs after tests are recorded. Isn't that a breaking change, or point to a breaking change? The guideline should almost be the reverse (after GA) since our intent is to avoid breaking changes - there are situations where the recordings need to change, but it shouldn't be because of a breaking change.
Additional comments:
- When should we be running live tests? We should add a guideline for this - at least before every GA release, and use different service API versions.
- Maybe we should establish guidelines on when to mock, with the default to not mock. e.g. fault injection appears to be one area where we heavily rely on mocks
- Who needs to decide what "appropriate" code coverage is actually reasonable. There should be a test for every single publically exposed operation. Also, error handling needs to be tested. Focus on clients rather than models. Payloads that can be serialized and deserialized should be tested.
- Reaching 100% code coverage is next to impossible - how do you select what to add to improve coverage.
- One guidelines that might be added regarding recordings is that perhaps you should exclude sensitive information from the recording.
@JonathanGiles - please add guidelines on recommendations on how to implement mockable clients. The current guidance is a copy of the general guidelines and not sufficient.