Allow running integration tests in a local VM with a local version of the Elastic stack

[cmacknz]

Building, uploading, and provisioning cloud resources works but takes significant time. See https://github.com/elastic/elastic-agent/issues/2842 and https://github.com/elastic/elastic-agent/issues/2856 for examples.

For developing and debugging tests, we should have a much faster local test setup and ideally be able to run all tests locally. Examples of how we could accomplish this:

1. Bring up the Elastic Stack in containers using elastic-package which already does exactly this for integrations.
2. Allow installing the agent in a local VM, defined with Vagrant or a similar tool.

The goal is not to cover the entire support matrix or architecture combination, but to optimize for a local development experience that is much faster than what we can achieve with a cloud based solution. We could limit this mode of operation to running in VMs for the latest versions of Ubuntu and Windows.

[fearful-symmetry]

Most of the tests require a root environment to run elastic-agent install. We could experiment with doing this in containers, but a more flexible approach would probably be something like vagrant. This would also allow us to develop something fairly flexible, as we can test on alternate OSes, etc. However, it would be a bit more work to stand up, since we don't have much existing vagrant client code. Quickly testing across multiple distros, OSes, etc, is something we're sorely lacking, and building out an integration runner with vagrant might allow us to kill two birds with one stone.

[cmacknz]

The fastest way to support this initially might be to just start everything as a container the way elastic-package does. This won't be exactly the same as what the tests in the cloud do, but it will test the actual e2e integration of the entire system. Using Docker has the advantage that this would also work on ARM Macs. If/when we do support VMs we likely want to use VMWare Fusion as the hypervisor since that officially supports M1 Macs today.

We need to support running the agent in a container anyway, see https://github.com/elastic/elastic-agent/issues/2900.

[cmacknz]

Using https://github.com/elastic/elastic-package to start everything definitely seems like the easiest path forward here.

[cmacknz]

Related, Blake introduced support for multipass to allow running tests against local VMs https://github.com/elastic/elastic-agent/pull/3202

[cmacknz]

I think the multipass support solves the need for running tests in local VMs.

The only other thing we could do is provision the stack locally using elastic-package instead of relying on cloud deployments to make the entire setup entirely local.

[pazone]

@cmacknz We experimented with elastic-package stack together with @pchila . It did work for us. Just need to wrap it up