search

fleetdm / fleet

Open-source platform for IT, security, and infrastructure teams. (Linux, macOS, Chrome, Windows, cloud, data center)
https://fleetdm.com
Other
3.11k stars 427 forks source link

Release QA: 4.57.0 #22111

Closed lukeheath closed 1 month ago

lukeheath commented 1 month ago

Goal: easy-to-follow test steps for checking a release manually

Important reference data

  1. fleetctl preview setup
  2. permissions documentation
  3. premium tests require license key (needs renewal) fleetctl preview --license-key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJGbGVldCBEZXZpY2UgTWFuYWdlbWVudCBJbmMuIiwiZXhwIjoxNjQwOTk1MjAwLCJzdWIiOiJkZXZlbG9wbWVudCIsImRldmljZXMiOjEwMCwibm90ZSI6ImZvciBkZXZlbG9wbWVudCBvbmx5IiwidGllciI6ImJhc2ljIiwiaWF0IjoxNjIyNDI2NTg2fQ.WmZ0kG4seW3IrNvULCHUPBSfFdqj38A_eiXdV_DFunMHechjHbkwtfkf1J6JQJoDyqn8raXpgbdhafDwv3rmDw
  4. premium tests require license key (active - Expires Sunday, January 1, 2023 12:00:00 AM) fleetctl preview --license-key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJGbGVldCBEZXZpY2UgTWFuYWdlbWVudCBJbmMuIiwiZXhwIjoxNjcyNTMxMjAwLCJzdWIiOiJGbGVldCBEZXZpY2UgTWFuYWdlbWVudCIsImRldmljZXMiOjEwMCwibm90ZSI6ImZvciBkZXZlbG9wbWVudCBvbmx5IiwidGllciI6InByZW1pdW0iLCJpYXQiOjE2NDI1MjIxODF9.EGHQjIzM73YyMbnCruswzg360DEYCsDi9uz48YcDwQHq90BabGT5PIXRiculw79emGj5sk2aKgccTd2hU5J7Jw

Smoke Tests

Smoke tests are limited to core functionality and serve as a pre-release final review. If smoke tests are failing, a release cannot proceed.

Fleet core:

Fleet version (Head to the "My account" page in the Fleet UI or run fleetctl version):

Web browser (e.g. Chrome 88.0.4324):

Prerequisites

  1. fleetctl preview is set up and running the desired test version using --tag parameters.
  2. Unless you are explicitly testing older browser versions, browser is up to date.
  3. Certificate & flagfile are in place to create new host.
  4. In your browser, clear local storage using devtools.
Test nameStep instructionsExpected resultpass/fail
$Name{what a tester should do}{what a tester should see when they do that}pass/fail
Update flow 1. remove all fleet processes/agents/etc using `fleetctl preview reset` for a clean slate 2. run `fleetctl preview` with no tag for latest stable 3. create a host/query to later confirm upgrade with 4. STOP fleet-preview-server instances in containers/apps on Docker 5. run `fleetctl preview` with appropriate testing tag All previously created hosts/queries are verified to still existpass/fail
Login flow 1. navigate to the login page and attempt to login with both valid and invalid credentials to verify some combination of expected results. 2. navigate to the login page and attempt to login with both valid and invalid sso credentials to verify expected results. 1. text fields prompt when blank 2. correct error message is "authentication failed" 3. forget password link prompts for email 4. valid credentials result in a successful login. 5. valid sso credentials result in a successful loginpass/fail
Query flowCreate, edit, run, and delete queries. 1. permissions regarding creating/editing/deleting queries are up to date with documentation 2. syntax errors result in error messaging 3. queries can be run manually pass/fail
Host FlowVerify a new host can be added and removed following modal instructions using your own device. 1. Host is added via command line 2. Host serial number and date added are accurate 3. Host is not visible after it is deleted 4. Warning and informational modals show when expected and make sense pass/fail
Packs flowVerify management, operation, and logging of ["2017 packs"](https://fleetdm.com/handbook/company/why-this-way#why-does-fleet-support-query-packs). 1. Packs successfully run on host machines after migrations 2. New Packs can be created 3. Packs can be edited and deleted 4. Packs results information is logged pass/fail
Log destination flowVerify log destination for software, query, policy, and packs. 1. Software, query, policy, and packs logs are successfully sent to external log destinations 2. Software, query, policy, and packs logs are successfully sent to Filesystem log destinations pass/fail
My device pageVerify the end user's my device page loads successfully. 1. Clicking the Fleet desktop item, then "My device" successfully loads the my device page. 2. The "My device" page is populated correctly and as expected. 3. Styling and padding appears correct. pass/fail
MDM enrollment flowVerify MDM enrollments, run MDM commands 1. Erase an ADE-eligible macOS host and verify able to complete automated enrollment flow. 2. With Windows MDM turned On, enroll a Windows host and verify MDM is turned On for the host. 3. Verify able to run MDM commands on both macOS and Windows hosts from the CLI. pass
MDM migration flowVerify MDM migration for ADE and non-ADE hosts 1. Turn off MDM on an ADE-eligible macOS host and verify that the native, "Device Enrollment" macOS notification appears. 2. On the My device page, follow the "Turn on MDM" instructions and verify that MDM is turned on. 3. Turn off MDM on a non ADE-eligible macOS host. 4. On the My device page, follow the "Turn on MDM" instructions and verify that MDM is turned on. pass
ScriptsVerify script library and execution 1. Verify able to run a script on all host types from CLI. 2. Verify scripts library upload/download/delete. 3. From Host details (Windows and macOS) run a script that should PASS, verify. 4. From Host details (Windows and macOS) run a script that should FAIL, verify. 5. Verify UI loading state and statuses for scripts. 6. Disable scripts globally and verify unable to run. 7. Verify scripts display correctly in Activity feed. pass
SoftwareVerify software library and install / download 1. Verify software library upload/download/delete. 2. From Host details (Windows and macOS) run an install that should PASS, verify. 3. From My Device (Windows and macOS) software tab should have self-service items available, verify. 4. Verify UI loading state and statuses for installing software. 6. Verify software installs display correctly in Activity feed. pass
OS settingsVerify OS settings functionality 1. Verify able to configure Disk encryption. 2. Verify host enrolled with Disk encryption enforced successfully encrypts. 3. Verify Profiles upload/download/delete (macOS & Windows). 4. Verify profiles are delivered to host and applied. pass
Setup experienceVerify macOS Setup experience 1. Configure End user authentication. 2. Upload a Bootstrap package. 3. Enroll an ADE-eligible macOS host and verify successful authentication. 4. Verify Bootstrap package is delivered. pass
OS updatesVerify OS updates flow 1. Configure OS updates (macOS & Windows). 2. Verify on-device that Nudge prompt appears (macOS). pass
Certificates UploadAPNs cert and ABM token renewal workflow 1. Renew APNs Certificate. 2. Renew ABM Token. 3. Ensure ADE host can enroll. pass/
Migration TestVerify Fleet can migrate to the next version with no issues. Using the migration scripts located in fleet/test/upgrade/ 1. Run the upgrade_test.go script using the most recent stable version of Fleet and `main`. 2. Upgrade test returns an 'OK' response. pass/fail
Release blockersVerify there are no outstanding release blocking tickets. 1. Check [this](https://github.com/fleetdm/fleet/labels/~release%20blocker) filter to view all open `~release blocker` tickets. 2. If any are found raise an alarm in the `#help-engineering` and `#g-mdm` (or `#g-endpoint-ops`) channels. pass/fail

Notes

Issues found new to this version:

Issues found that reproduce in last stable version:

What has not been tested:

Include any notes on whether issues should block release or not as needed:

fleetd agent:

Includes updates to:

List versions changes for any component updates below:

Prerequisites

  1. Build a new fleetd from the release candidate branch as neded for Orbit, Desktop, and Chrome Extension.
Test nameStep instructionsExpected resultpass/fail
$Name{what a tester should do}{what a tester should see when they do that}pass/fail
`fleetd` tests 1. Create binaries for Mac, Windows, and Ubuntu running against the `edge` channels and install (--orbit-channel edge, --desktop-channel edge).
2. Work with engineer leading the release to push changes to the `edge` channel. 		1. Confirm the hosts running on the edge channel receive the update and are working correctly.
2. Confirm any new features and/or bug fixes associated with this release are working as intended. 		pass/fail
`fleetd` auto-update tests 1. Conduct the [`fleetd` auto-update n+1 test]([url](https://github.com/fleetdm/fleet/blob/main/tools/tuf/test/Fleetd-auto-update-test-guide.md))
2. QA certifies new release by commenting in issue.
3. Engineer waits at least 1 business day, then promotes update to `stable`. 		1. Agent successfully auto-updates.
2. Issue is certified by QA.
3. Agent is promoted to `stable`.
4. Confirms agents running on `stable` receive the new update. 		pass/fail

Notes

Issues found new to this version:

Issues found that reproduce in last stable version:

What has not been tested:

Include any notes on whether issues should block release or not as needed:

lucasmrod commented 1 month ago

Endpoints OPS gives 👍

PezHub commented 1 month ago

MDM is gtg 👍

PezHub commented 1 month ago

Migration results look good (yes the ver order is all over the place 🙃 )

~/fleetdm/fleet minor-fleet-v4.57.0 ❯ FLEET_VERSION_A=v4.45.0 FLEET_VERSION_B=minor-fleet-v4.57.0 go test ./test/upgrade ok github.com/fleetdm/fleet/v4/test/upgrade 45.501s ~/fleetdm/fleet minor-fleet-v4.57.0 ❯ FLEET_VERSION_A=v4.46.0 FLEET_VERSION_B=minor-fleet-v4.57.0 go test ./test/upgrade ok github.com/fleetdm/fleet/v4/test/upgrade 35.661s ~/fleetdm/fleet minor-fleet-v4.57.0 ❯ FLEET_VERSION_A=v4.47.0 FLEET_VERSION_B=minor-fleet-v4.57.0 go test ./test/upgrade ok github.com/fleetdm/fleet/v4/test/upgrade 37.672s ~/fleetdm/fleet minor-fleet-v4.57.0 ❯ FLEET_VERSION_A=v4.48.0 FLEET_VERSION_B=minor-fleet-v4.57.0 go test ./test/upgrade ok github.com/fleetdm/fleet/v4/test/upgrade 35.003s ~/fleetdm/fleet minor-fleet-v4.57.0 ❯ FLEET_VERSION_A=v4.54.0 FLEET_VERSION_B=minor-fleet-v4.57.0 go test ./test/upgrade ok github.com/fleetdm/fleet/v4/test/upgrade 34.735s ~/fleetdm/fleet minor-fleet-v4.57.0 ❯ FLEET_VERSION_A=v4.54.1 FLEET_VERSION_B=minor-fleet-v4.57.0 go test ./test/upgrade ok github.com/fleetdm/fleet/v4/test/upgrade 43.745s ~/fleetdm/fleet minor-fleet-v4.57.0 ❯ FLEET_VERSION_A=v4.55.0 FLEET_VERSION_B=minor-fleet-v4.57.0 go test ./test/upgrade ok github.com/fleetdm/fleet/v4/test/upgrade 37.721s ~/fleetdm/fleet minor-fleet-v4.57.0 ❯ FLEET_VERSION_A=v4.55.1 FLEET_VERSION_B=minor-fleet-v4.57.0 go test ./test/upgrade ok github.com/fleetdm/fleet/v4/test/upgrade 36.810s ~/fleetdm/fleet minor-fleet-v4.57.0 ❯ FLEET_VERSION_A=v4.55.2 FLEET_VERSION_B=minor-fleet-v4.57.0 go test ./test/upgrade ok github.com/fleetdm/fleet/v4/test/upgrade 39.400s ~/fleetdm/fleet minor-fleet-v4.57.0 ❯ FLEET_VERSION_A=v4.54.2 FLEET_VERSION_B=minor-fleet-v4.57.0 go test ./test/upgrade ok github.com/fleetdm/fleet/v4/test/upgrade 38.400s ~/fleetdm/fleet minor-fleet-v4.57.0 ❯ FLEET_VERSION_A=v4.53.0 FLEET_VERSION_B=minor-fleet-v4.57.0 go test ./test/upgrade ok github.com/fleetdm/fleet/v4/test/upgrade 38.723s ~/fleetdm/fleet minor-fleet-v4.57.0 ❯ FLEET_VERSION_A=v4.53.1 FLEET_VERSION_B=minor-fleet-v4.57.0 go test ./test/upgrade ok github.com/fleetdm/fleet/v4/test/upgrade 37.776s ~/fleetdm/fleet minor-fleet-v4.57.0 ❯ FLEET_VERSION_A=v4.56.0 FLEET_VERSION_B=minor-fleet-v4.57.0 go test ./test/upgrade ok github.com/fleetdm/fleet/v4/test/upgrade 35.443s

fleet-release commented 1 month ago

Fleet's manual checks, smooth like river's steady flow, Trust in code's progress.