[Obs] Annotations requirements + SLO POC

We want to allow users to store annotations in our system so we can display those on charts across observability. Annotations are already used in some places in APM, tied to services. We want to expand this to other areas of observability, starting with SLOs.

Acceptance criteria

This ticket should be considered "done" when we have:

An initial set of requirements for this observability annotations API, as we intend to use it in places beyond its current usage in APM. Specifically, these requirements should focus on the SLO use-case (led by @grabowskit), with some consideration of the near-future plans that come from our RCA initiative (led by @drewpost).
A working POC demonstrating these requirements in action, being used in the SLO product

Notes

The observability annotation API endpoint is found at /api/observability/annotation
I'm not sure if documentation exists for this endpoint
The observability plugin also exposes an annotations client which can be used in plugin JS
Additionally, there is an APM-specific version of the annotations api, which uses the observability annotations client under the hood but accepts a more restricted set of inputs meant for APM use.
Tests that demonstrate how this API can be used
Type definition for what's accepted when creating an "Annotation"

# Annotation type, taken from link above
interface Annotation {
  annotation: {
    type: string;
  };
  tags?: string[];
  message: string;
  service?: {
    name?: string;
    environment?: string;
    version?: string;
  };
  event: {
    created: string;
  };
  '@timestamp': string;
}

Open questions

(Shahzad) Should annotations show in every chart in observability, or should they be tied to some "scope" or "context"?
(Shahzad) Do we want to develop permission model around annotations per space etc?
(Shahzad) Do we want to limit them by some data type? let's say an annotation attached to a service name will only display in that service charts?
(Shahzad) What kind of other data definitions we will need to supports, service name? monitor name? host etc?
Are annotations only to be created via UI or should the API be public and allow code to create annotations via REST?
If we allow these to be created via code, using a public REST API, do we need to think about authentication on that API? Assuming we will want to allow something like a GitHub webhook to create annotations, or JIRA to push events in as annotations, etc, how do we allow this while avoiding someone's production system being flooded with fake annotations created by a bad actor?
Are there any other fields missing in the current Annotation type definition and document mapping?
Are there any other API methods missing in the current API specification?
Are we missing other big questions to think about up front (even if we don't fully address them all in the initial work)?

elastic / kibana

[Obs] Annotations requirements + SLO POC #182429

Acceptance criteria

Notes

Open questions