Open mjcheetham opened 1 year ago
As mentioned by @derrickstolee in #1045 we should make sure we include an end-to-end test to validate TRACE2
event/normal output as part of this effort.
Potential scenarios to cover: | scenario | test type | provider(s) | platform(s) | credstore(s) | framework |
---|---|---|---|---|---|---|
get with existing credential | integration | Bitbucket, GitHub, GitLab, Generic | macOS, Linux, Windows, WSL | Keychain, Credential Manager, Secret Service, GPG/Pass | Cucumber | |
get without existing credential | integration | Bitbucket, GitHub, GitLab, Generic | macOS, Linux, Windows, WSL | Keychain, Credential Manager, Secret Service, GPG/Pass | Cucumber | |
store | integration | Bitbucket, GitHub, GitLab, Generic | macOS, Linux, Windows, WSL | Keychain, Credential Manager, Secret Service, GPG/Pass | Cucumber | |
erase | integration | Bitbucket, GitHub, GitLab, Generic | macOS, Linux, Windows, WSL | Keychain, Credential Manager, Secret Service, GPG/Pass | Cucumber | |
TRACE/TRACE2 output validation | integration | N/A | macOS, Linux, Windows, WSL | N/A | Cucumber | |
OAuth using GUI | end-to-end | Bitbucket, GitHub, GitLab | macOS, Linux, Windows, WSL | Keychain, Credential Manager, Secret Service, GPG/Pass | TBD | |
OAuth using Terminal | end-to-end | Bitbucket, GitHub, GitLab | macOS, Linux, Windows, WSL | Keychain, Credential Manager, Secret Service, GPG/Pass | Cucumber | |
Device code using GUI | end-to-end | Bitbucket, GitHub, GitLab | macOS, Linux, Windows, WSL | Keychain, Credential Manager, Secret Service, GPG/Pass | TBD | |
Device code using Terminal | end-to-end | Bitbucket, GitHub, GitLab | macOS, Linux, Windows, WSL | Keychain, Credential Manager, Secret Service, GPG/Pass | Cucumber | |
PAT using GUI | end-to-end | Bitbucket, GitHub, GitLab | macOS, Linux, Windows, WSL | Keychain, Credential Manager, Secret Service, GPG/Pass | TBD | |
PAT using Terminal | end-to-end | Bitbucket, GitHub, GitLab | macOS, Linux, Windows, WSL | Keychain, Credential Manager, Secret Service, GPG/Pass | Cucumber | |
Username/password using GUI | end-to-end | Bitbucket, GitLab, Generic | macOS, Linux, Windows, WSL | Keychain, Credential Manager, Secret Service, GPG/Pass | TBD | |
Username/password using Terminal | end-to-end | Bitbucket, GitLab, Generic | macOS, Linux, Windows, WSL | Keychain, Credential Manager, Secret Service, GPG/Pass | Cucumber |
Candidates for UI testing:
There have been several bugs over the lifetime of GCM that have occurred around installation or platform integration that were not caught in unit or manual testing. To avoid such issues going forward we propose creating a CI workflow, for all supported platforms, that runs a basic suite of integration or end-to-end tests including installation, basic authentication flows, and uninstallation.
We should be able to achieve this using GitHub hosted runners and a matrix workflow that runs the appropriate installer and performs simple authentication actions.
There are two options (not mutually exclusive) for handling UI prompts: