search

git-ecosystem / git-credential-manager

Secure, cross-platform Git credential storage with authentication to GitHub, Azure Repos, and other popular Git hosting services.
Other
6.59k stars 1.71k forks source link

Create end-to-end testing framework and CI workflows #1103

Open mjcheetham opened 1 year ago

mjcheetham commented 1 year ago

There have been several bugs over the lifetime of GCM that have occurred around installation or platform integration that were not caught in unit or manual testing. To avoid such issues going forward we propose creating a CI workflow, for all supported platforms, that runs a basic suite of integration or end-to-end tests including installation, basic authentication flows, and uninstallation.

We should be able to achieve this using GitHub hosted runners and a matrix workflow that runs the appropriate installer and performs simple authentication actions.

There are two options (not mutually exclusive) for handling UI prompts:

  1. Replace the UI components with a 'test' UI helper that is drivable by the test harness.
  2. Use a tool like AutoHotkey (Windows only) on platforms that have a free and easy to use UI automation tool.
ldennington commented 1 year ago

As mentioned by @derrickstolee in #1045 we should make sure we include an end-to-end test to validate TRACE2 event/normal output as part of this effort.

ldennington commented 1 year ago
Potential scenarios to cover: scenario test type provider(s) platform(s) credstore(s) framework
get with existing credential integration Bitbucket, GitHub, GitLab, Generic macOS, Linux, Windows, WSL Keychain, Credential Manager, Secret Service, GPG/Pass Cucumber
get without existing credential integration Bitbucket, GitHub, GitLab, Generic macOS, Linux, Windows, WSL Keychain, Credential Manager, Secret Service, GPG/Pass Cucumber
store integration Bitbucket, GitHub, GitLab, Generic macOS, Linux, Windows, WSL Keychain, Credential Manager, Secret Service, GPG/Pass Cucumber
erase integration Bitbucket, GitHub, GitLab, Generic macOS, Linux, Windows, WSL Keychain, Credential Manager, Secret Service, GPG/Pass Cucumber
TRACE/TRACE2 output validation integration N/A macOS, Linux, Windows, WSL N/A Cucumber
OAuth using GUI end-to-end Bitbucket, GitHub, GitLab macOS, Linux, Windows, WSL Keychain, Credential Manager, Secret Service, GPG/Pass TBD
OAuth using Terminal end-to-end Bitbucket, GitHub, GitLab macOS, Linux, Windows, WSL Keychain, Credential Manager, Secret Service, GPG/Pass Cucumber
Device code using GUI end-to-end Bitbucket, GitHub, GitLab macOS, Linux, Windows, WSL Keychain, Credential Manager, Secret Service, GPG/Pass TBD
Device code using Terminal end-to-end Bitbucket, GitHub, GitLab macOS, Linux, Windows, WSL Keychain, Credential Manager, Secret Service, GPG/Pass Cucumber
PAT using GUI end-to-end Bitbucket, GitHub, GitLab macOS, Linux, Windows, WSL Keychain, Credential Manager, Secret Service, GPG/Pass TBD
PAT using Terminal end-to-end Bitbucket, GitHub, GitLab macOS, Linux, Windows, WSL Keychain, Credential Manager, Secret Service, GPG/Pass Cucumber
Username/password using GUI end-to-end Bitbucket, GitLab, Generic macOS, Linux, Windows, WSL Keychain, Credential Manager, Secret Service, GPG/Pass TBD
Username/password using Terminal end-to-end Bitbucket, GitLab, Generic macOS, Linux, Windows, WSL Keychain, Credential Manager, Secret Service, GPG/Pass Cucumber

Candidates for UI testing: