We need to level up our agent's testing

[directionless]

Today, our agent test process is a bit ad hoc. It's not bad, but it could be better. Over time, some ideas that have come up:

• Acceptance testing for new launcher and osquery releases
• how to test autoupdate
• exercising tables
• With the new Device Trust direction, how can we test that flow?

I suspect this is going to have a lot of devops flavored work. Though much of it could be written in go.

[RebeccaMahany]

TODOs:

• [ ] Run existing basic suite on a nightly basis: https://github.com/kolide/monorepo/pull/114
• [ ] Write more tests in the basic suite -- exercise localserver functionality, add performance checks
• [ ] Add a new suite to test installation and enrollment for nightly version -- requires kworker to build an on-demand package for latest nightly: https://github.com/kolide/monorepo/pull/116
• [ ] Add and use more VM templates for different Linux flavors
• [ ] BYOL to test Windows client versions
• [ ] Run tests on macOS: use an AWS EC2 instance?
• [ ] Support ability to run suites against a given launcher branch
• [ ] Something that can cover UI tests (notifications, menu bar app, browser opening, performance while desktop is running, device trust flow)

[directionless]

Run tests on macOS: use an AWS EC2 instance?

I wonder if there's an option to launch our own VMs on one of our macstadium machines, or a machine under a desk somewhere. That would limit us to m1, but that might be okay