[Observability Onboarding] Catch 22 when onboarding to APM (?)

dkarlovi commented 5 months ago

Kibana version:

8.13.0

Elasticsearch version:

8.13.0

Server OS version:

Docker

Browser version:

Google Chrome 124.0.6367.118 (Official Build) (64-bit)

Browser OS version:

Fedora Linux 40

Original install method (e.g. download page, yum, from source, etc.):

Docker image

Describe the bug:

When onboarding to APM, there seems to be a catch 22 of sorts:

Kibana needs the APM server to be available and running
APM server needs the index templates, which Kibana installs

Steps to reproduce:

# docker-compose.yaml
services:
    search:
        image: elasticsearch:8.13.0
        environment:
            - cluster.name=es1
            - discovery.type=single-node
            - xpack.security.enabled=false
            - ES_JAVA_OPTS=-Xms1g -Xmx1g
    kibana:
        image: kibana:8.13.0
        environment:
            - ELASTICSEARCH_HOSTS=http://search:9200
            - ELASTIC_APM_SERVER_URL=http://telemetry:8200
    telemetry:
        image: elastic/apm-server:8.13.0
        command: apm-server -e -E output.elasticsearch.hosts=http://search:9200

When I use "Check APM Server status" on /kibana/app/home#/tutorial/apm

I get:

No APM Server detected. Please make sure it is running and you have updated to 7.0 or higher.

but, from the APM server logs, I keep getting

{"log.level":"error","@timestamp":"2024-05-03T15:04:15.720Z","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.waitReady","file.name":"beater/waitready.go","file.line":62},"message":"precondition 'apm integration installed' failed: error querying Elasticsearch for integration index templates: unexpected HTTP status: 404 Not Found ({\"error\":{\"root_cause\":[{\"type\":\"resource_not_found_exception\",\"reason\":\"index template matching [metrics-apm.service_summary.60m] not found\"}],\"type\":\"resource_not_found_exception\",\"reason\":\"index template matching [metrics-apm.service_summary.60m] not found\"},\"status\":404}): to remediate, please install the apm integration: https://ela.st/apm-integration-quickstart","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-05-03T15:04:20.720Z","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.waitReady","file.name":"beater/waitready.go","file.line":62},"message":"precondition 'apm integration installed' failed: error querying Elasticsearch for integration index templates: unexpected HTTP status: 404 Not Found ({\"error\":{\"root_cause\":[{\"type\":\"resource_not_found_exception\",\"reason\":\"index template matching [metrics-apm.service_summary.60m] not found\"}],\"type\":\"resource_not_found_exception\",\"reason\":\"index template matching [metrics-apm.service_summary.60m] not found\"},\"status\":404}): to remediate, please install the apm integration: https://ela.st/apm-integration-quickstart","service.name":"apm-server","ecs.version":"1.6.0"}

From my understanding reading a bunch of ES docs, forums and error messages, Kibana will install these templates (as evidenced in the errors), but Kibana doesn't seem to think the APM server is running, even though it is and it's accessible from Kibana's container:

kibana@e5b74be5283d:~$ env | grep SERVER_URL
ELASTIC_APM_SERVER_URL=http://telemetry:8200
kibana@e5b74be5283d:~$ curl http://telemetry:8200
{
  "build_date": "2024-03-21T15:24:03Z",
  "build_sha": "77c62777a39b536ff3a2a663456a2e0086787552",
  "publish_ready": false,
  "version": "8.13.0"
}

Expected behavior:

Kibana sees the APM server and installs the templates (or installs the templates even if it cannot see the APM server?)

Alternatively, Kibana explains what it tried to do when checking and provide a hint how the APM server could be made available (for example, if it's a configuration issue, what APM server endpoint did it try and how to change that).

Screenshots (if relevant):

Home-Elastic