[APM] Serverless - Cannot visualize annotations in serverless

[dominiqueclarke]

Original install method (e.g. download page, yum, from source, etc.): Serverless production

Description of the problem including expected versus actual behavior:

In serverless, my annotations do not appear in APM visualizations. I tested with viewer, editor, and admin roles. Here I'm looking at the last 14 days of data without annotations.

Checking the index shows that I do have quite a few annotations for the last 14 days.

[
      {
        "_index": "observability-annotations",
        "_id": "_8LUmJABnUF2DueOopNO",
        "_score": 1,
        "_source": {
          "message": "Deployment 1.2",
          "@timestamp": "2024-07-01T15:45:58.000Z",
          "service": {
            "version": "1.2",
            "name": "synth-node-2"
          },
          "annotation": {
            "type": "deployment"
          },
          "tags": [
            "apm"
          ],
          "event": {
            "created": "2024-07-09T18:49:49.513Z"
          }
        }
      },
      {
        "_index": "observability-annotations",
        "_id": "tsLXmJABnUF2DueOqJWT",
        "_score": 1,
        "_source": {
          "message": "Deployment 1.2",
          "@timestamp": "2024-07-01T15:45:58.000Z",
          "service": {
            "version": "1.2",
            "name": "synth-node-2"
          },
          "annotation": {
            "type": "deployment"
          },
          "tags": [
            "apm"
          ],
          "event": {
            "created": "2024-07-09T18:53:08.626Z"
          }
        }
      },
      {
        "_index": "observability-annotations",
        "_id": "acLYmJABnUF2DueOy5b7",
        "_score": 1,
        "_source": {
          "message": "Deployment 1.2",
          "@timestamp": "2024-07-01T15:45:58.000Z",
          "service": {
            "version": "1.2",
            "name": "synth-node-2"
          },
          "annotation": {
            "type": "deployment"
          },
          "tags": [
            "apm"
          ],
          "event": {
            "created": "2024-07-09T18:54:23.226Z"
          }
        }
      },
      {
        "_index": "observability-annotations",
        "_id": "f8LYmJABnUF2DueO95Y8",
        "_score": 1,
        "_source": {
          "message": "Deployment 1.2",
          "@timestamp": "2024-07-01T15:45:58.000Z",
          "service": {
            "version": "1.2",
            "name": "synth-node-2"
          },
          "annotation": {
            "type": "deployment"
          },
          "tags": [
            "apm"
          ],
          "event": {
            "created": "2024-07-09T18:54:34.298Z"
          }
        }
      },
      {
        "_index": "observability-annotations",
        "_id": "qsLZmJABnUF2DueOPpa5",
        "_score": 1,
        "_source": {
          "message": "Deployment 1.2",
          "@timestamp": "2024-07-01T15:45:58.000Z",
          "service": {
            "version": "1.2",
            "name": "synth-node-2"
          },
          "annotation": {
            "type": "deployment"
          },
          "tags": [
            "apm"
          ],
          "event": {
            "created": "2024-07-09T18:54:52.600Z"
          }
        }
      },
      {
        "_index": "observability-annotations",
        "_id": "BcLZmJABnUF2DueO2pdq",
        "_score": 1,
        "_source": {
          "message": "Deployment 1.2",
          "@timestamp": "2024-07-01T15:45:58.000Z",
          "service": {
            "version": "1.2",
            "name": "synth-node-2"
          },
          "annotation": {
            "type": "deployment"
          },
          "tags": [
            "apm"
          ],
          "event": {
            "created": "2024-07-09T18:55:32.457Z"
          }
        }
      },
      {
        "_index": "observability-annotations",
        "_id": "MsLfmJABnUF2DueO05rX",
        "_score": 1,
        "_source": {
          "message": "Deployment 1.2",
          "@timestamp": "2024-07-01T15:45:58.000Z",
          "service": {
            "version": "1.2",
            "name": "synth-node-2"
          },
          "annotation": {
            "type": "deployment"
          },
          "tags": [
            "apm"
          ],
          "event": {
            "created": "2024-07-09T19:02:03.990Z"
          }
        }
      },
      {
        "_index": "observability-annotations",
        "_id": "WcLsmJABnUF2DueOeaEp",
        "_score": 1,
        "_source": {
          "message": "Deployment 1.2",
          "@timestamp": "2024-07-01T15:45:58.000Z",
          "service": {
            "version": "1.2",
            "name": "synth-node-2"
          },
          "annotation": {
            "type": "deployment"
          },
          "tags": [
            "apm"
          ],
          "event": {
            "created": "2024-07-09T19:15:52.744Z"
          }
        }
      },
      {
        "_index": "observability-annotations",
        "_id": "D8LvmJABnUF2DueOmKN1",
        "_score": 1,
        "_source": {
          "message": "Deployment 1.2",
          "@timestamp": "2024-07-09T15:45:58.000Z",
          "service": {
            "version": "1.2",
            "name": "synth-node-2"
          },
          "annotation": {
            "type": "deployment"
          },
          "tags": [
            "apm"
          ],
          "event": {
            "created": "2024-07-09T19:19:17.364Z"
          }
        }
      },
      {
        "_index": "observability-annotations",
        "_id": "k8LwmJABnUF2DueOgKOx",
        "_score": 1,
        "_source": {
          "message": "Deployment 1.2",
          "@timestamp": "2024-07-09T03:45:58.000Z",
          "service": {
            "version": "1.2",
            "name": "synth-node-2"
          },
          "annotation": {
            "type": "deployment"
          },
          "tags": [
            "apm"
          ],
          "event": {
            "created": "2024-07-09T19:20:16.815Z"
          }
        }
      }
    ]

Steps to reproduce:

1. Create a new serverless project and onboard some APM data
2. Create an api key with an admin user
3. Send an annotations API request. Example ```

   curl -X POST https://{kibanaEndpoint}/api/apm/services/{serviceName}/annotation -H 'Content-Type: application/json' -H 'kbn-xsrf: true' -H 'Authorization: ApiKey {apiKey}' -H 'elastic-api-version: 2023-10-31' -d '{ "@timestamp": "2024-07-01T15:45:58Z", "service": { "version": "1.2" }, "message": "Deployment 1.2" 

   4. Navigate to that service's detail page and onto the transactions tab. Notice that annotations do not appear

[elasticmachine]

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)