[Observability] [Annotation] Creating annotation takes long time on Serverless

Creating annotation on Serverless with API or UI takes a long time (around ~15 mins). Also, most of the times, API fails with "backend closed connection" error.

In the below example, the annotation request was made at 08:33:23, but the annotation was created at 08:50:26.

  {
        "_index": "observability-annotations",
        "_id": "Yes4ipIBdJudItRKu7oC",
        "_score": 1,
        "_source": {
          "message": "New version release for cartservice: 2.0.0",
          "@timestamp": "2024-10-14T08:33:23.000Z",
          "service": {
            "version": "2.0.0",
            "name": "cartservice"
          },
          "annotation": {
            "type": "deployment",
            "title": "New version release for cartservice: 2.0.0"
          },
          "tags": [
            "apm"
          ],
          "event": {
            "created": "2024-10-14T08:50:26.688Z"
          }
        }
      }

elastic / kibana

[Observability] [Annotation] Creating annotation takes long time on Serverless #196067