Create a PoC integration test using testcontainers

[mdelapenya]

Starting up the agent container and checking that there is 4 processes running (2metricbeat 2filebeat) seems to be a good start right?

Originally posted by @jlind23 in https://github.com/elastic/elastic-agent/issues/56#issuecomment-1083141472

The idea is to use testcontainers-go: https://github.com/testcontainers/testcontainers-go/

[mdelapenya]

@jlind23 I've thinking about this specific test:

Starting up the agent container and checking that there is 4 processes running (2metricbeat 2filebeat)

I consider it not to be a good candidate for the integration test suite because it implies a lot of things:

• we do not know how the agent was installed: the elastic-agent image, or a docker container with a TAR file installed on it
• the enrollment of the agent using a fleet-server
• the default package policy for the agent has been used telling the agent to start filebeat and metricbeat.

And this specific test is already implemented as an e2e test here: https://github.com/elastic/e2e-testing/blob/b99a810881793dd819caf1ed47de0c121ac3be76/e2e/_suites/fleet/features/backend_processes.feature#L6-L10

But for me, the most important thing to not consider it a valid integration test is that it requires to package the agent, in any way: docker, tar, deb, rpm... And this happens too late in the development life cycle.

I've run time PACKAGES=docker mage package to only build the docker images and it takes:

real    47m26.519s
user    4m6.523s
sys     2m15.288s

This makes me think how the team is testing the features: do they package the agent every time to have a working binary? I'd have expected to simply run go run main.go, even using the IDE debugger while developing or running the unit/integration tests, and I guess this is the common expectation from Go developers with the intention to contribute to the project.

I'd advocate to have an integration test suite that:

• fetches the beats dependencies (filebeat, heartbeat, metricbeat), having them (and their specific versions) declared explicitly.
• runs the Go test with go test without the need to package the elastic-agent binaries. It would be fine use mage for it, but as simple as possible.
• "simulates/mocks" the runtime dependencies such as an elasticsearch instance, or fleet-server, or the system to monitor
• uses a library to start those dependencies, attaching their life cycle to the life cycle of the specific test that invoked them: ex. Test_A starts an elasticsearch instance at the beginning of the test, killing it right at the end of it.

[jlind23]

@cmacknz @ph I may need your help here it is something impactful for the developer experience around the Elastic Agent V2.

[jlind23]

Now that we have our own integration testing framework I think we are good to close this one as won't fix. cc @ycombinator