[Meta] Fleet Server detailed tests e2e-testing review and implementation

[EricDavisX]

We have Fleet Server usage in the e2e-testing repo, so it is covered in a sense (which is great!) - but this is not the same as having a detailed review of test coverage for the FS itself and a feature file that focuses on it.

Prior work and initial support was recorded here: https://github.com/elastic/e2e-testing/issues/438

Remaining work is:

• [ ] adjust e2e-testing framework scripting with a new feature file called 'Fleet Server' which will run the below validations for the Fleet Server binary...
  • [ ] testing it weekly on at least one each of the Windows, Linux, macOS
  • [ ] testing it as well on at least one of .deb or .rpm install
  • [ ] testing it in Docker as well (Agent side usage is not thorough enough coverage) Then, after environment configuration is covered also, write test cases scenarios for s
• [ ] stop and re-start of Fleet Server
• [ ] artifact download (new Integration that is not baked in to default Agent bundle)
• [ ] Error handling when mis-configured
• [ ] Error handling when ES is unavailable

That looks like most of the cases from here, at least it is a start - Engineering should review functional coverage and suggest more as needed: https://elastic.testrail.io/index.php?/suites/view/27&group_by=cases:section_id&group_order=asc&display_deleted_cases=0&group_id=25988

[EricDavisX]

@blakerouse @ph @aleksmaus @scunningham hi - just tagging you for reference. If anyone wants to shout out high level (or detailed!) tests we want to ensure are run for FS in an 'end to end' scenario vs Unit tests. Integration with CI support for PRs to run the e2e scenarios for FS is separate, fyi.

[EricDavisX]

Hi - I'm re-opening this. Thanks for the efforts in https://github.com/elastic/e2e-testing/pull/1628 Manu! I don't believe that exactly closes this though.

We still need to identify what specific tests we want to run for Fleet Server, apart from what is inherently covered in its usage in the Agent / Integrations side tests. We have a framework now, so that is great! But, if I'm not mistaken, the framework inherently uses Fleet Server in Docker, which is to restrictive from the perspective of FS, we need to implement a set of tests that make use of the Fleet Server on the new vm-hosts to get coverage over the binary on at least some citation of the major 3 OS variants (Windows, Linux, macOS) as well as coverage over .deb or .rpm install and running FS itself (which is not covered in the .deb/.rpm installs of Agent in current e2e-testing). As close as Docker usage is to our ESS/Cloud usage, our tests don't cover start / stop of the process or container for FS (only for non-Fleet-Server running Agents).

I recommend we be fairly careful in what we close and how we assess what work is needed, even when the ticket sucks and is vague (awaiting time from someone!). Needing input is still the state here, but I can lay out a little more depth in the full description, as translated from my notes here. I'll add this - it may actually just be bad luck, I think this is the 2nd time that general 'Fleet Server' automation testing was closed out.

@mdelapenya thanks again. @jlind23 fyi, a good issue to focus on for Agent Control Plane team.

[mdelapenya]

Adding @adam-stokes for awareness, as he can provide more inputs for the stack provisioning on VMs.

To address the need of running fleet-server tests for PRs, we need to figure out one thing first about the CI snapshots that we want to consume, as explained in the linked private issue.