[META] Suite Discovery / Monitor Management Integration

[andrewvc]

Suites, today, operate in a stateless manner, and have a number of UX shortcomings especially when it comes to error reporting (see https://github.com/elastic/beats/issues/27924). This issue is the longer term fix for https://github.com/elastic/uptime/issues/425.

The proposed fix here (originally @drewpost 's recommendation) is to use suites to control the monitor management objects in Uptime.

Implementation issues

• [x] https://github.com/elastic/uptime/issues/435
• [x] https://github.com/elastic/beats/issues/29917
• [x] https://github.com/elastic/beats/issues/30295
• [x] https://github.com/elastic/uptime/issues/438
• [x] https://github.com/elastic/kibana/issues/123426

Goals

The goal here is reduce the impedance mismatch between suites and inline monitors while still enabling a git-ops based workflow.

Personas

• Uptime user: Person using the uptime app, usually an administrator responsible for infrastructure apps run on
• Test Creator: Person responsible for writing / maintaining tests (could be a sysadmin, programmer, QA, etc.)

User Stories

This is a partial list of stories covered by this change

• As an Uptime user I would like to mute individual journeys in a suite through the Uptime UI when they fail due to a temporary condition.
• As a test creator I would like to add/remove/delete a test when desired
• As an Uptime user, when a suite fails to execute I would still like to see its individual journeys listed, and see its errors in a clear place.

Implementation

At a logical level we will need to decouple suite journey discovery from execution, leading to the following flow:

1. Suite refresh interval triggered
2. Suite is downloaded
3. Suite is unpacked and journeys discovered via --dry-run
4. Existing monitors no longer defined in the suite are deleted (possibly as a soft delete) in kibana management
5. Newly created monitors are added, existing ones are updated
6. Monitors created from this process cannot be deleted or updated, though they may be disabled.

In terms of code, consider the following straw-man approach:

1. When a zip url monitor is configured it now no executes journeys, but emits a manifest document to ES describing which journeys were discovered in the zip
2. A kibana background job checks for the latest manifest documents and uses them to update central monitor management
3. These newly configured monitors act as zip monitors currently do, filtering only for the journey they are set to run.

This would look something like:

The desirable properties of this approach are that:

• It doesn't require an excessive number of changes to our current stack
• It maintains the current security profile, since discovery of monitors requires arbitrary code execution it is simplest to do it in heartbeat still

Needs design etc. CC @liciavale

[vigneshshanmugam]

Had some conversations with Andrew regarding the approach listed here vs https://github.com/elastic/beats/issues/29709, Even though the manifest proposal explained here is a bit complex than the other one, it does have better control over the journeys/monitors that are created via suites. It solves both the personas listed above and allows more fine tuning on the journey level.

Needs solving

• Monitors created from the suite endpoints would be created as suite pointing to same zip_url and filtered for specific journey. So a single suite monitor(10 journeys) created via UI, would end up creating 10 monitors with filtered journey names extracted from the manifest. We need to cache the extracted zip files for the service to work smoothly, otherwise we would end up downloading them for every monitors that are created from the suites.

Options

• S3 or Cloud storage - This option is already on the table
• Kubernetes volumes - Have been exploring this idea in service before for running parallel execution of monitors across kubernetes pods. With this we can create a Persistent volume (NFS) in the Kubernetes cluster and then mount that shared volumes to all the pods that are running that suite. Think about them like a docker volume. We can control the policy of this persistent volume and delete them whenever the suite monitor is deleted.

[andrewvc]

Closing in favor of https://github.com/elastic/synthetics/issues/470