Setting up APM question

[duongpc]

Hello,

I followed the instruction (http://localhost:5601/app/home#/tutorial/apm) to install APM server

curl -L -O https://artifacts.elastic.co/downloads/apm-server/apm-server-8.1.0-amd64.deb
dpkg -i apm-server-8.1.0-amd64.deb

After that I run

service apm-server start

{"log.level":"info","@timestamp":"2022-05-11T01:58:52.663Z","log.origin":{"file.name":"instance/beat.go","file.line":670},"message":"Home path: [/usr/share/apm-server] Config path: [/etc/apm-server] Data path: [/var/lib/apm-server] Logs path: [/var/log/apm-server]","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-11T01:58:52.663Z","log.origin":{"file.name":"instance/beat.go","file.line":678},"message":"Beat ID: e43de2b9-5f3d-4fca-bb66-4b490b775333","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-11T01:58:52.663Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1048},"message":"Beat info","service.name":"apm-server","system_info":{"beat":{"path":{"config":"/etc/apm-server","data":"/var/lib/apm-server","home":"/usr/share/apm-server","logs":"/var/log/apm-server"},"type":"apm-server","uuid":"e43de2b9-5f3d-4fca-bb66-4b490b775333"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-11T01:58:52.663Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1057},"message":"Build info","service.name":"apm-server","system_info":{"build":{"commit":"7d34f3e579c7fadc5e70df76ed9b39332a40d0de","libbeat":"8.1.0","time":"2022-03-03T14:34:24.000Z","version":"8.1.0"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-11T01:58:52.663Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1060},"message":"Go runtime info","service.name":"apm-server","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":12,"version":"go1.17.6"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-11T01:58:52.664Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1064},"message":"Host info","service.name":"apm-server","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-05-11T00:02:22Z","containerized":true,"name":"c15db73e9a4a","ip":["127.0.0.1/8","172.17.0.2/16"],"kernel_version":"5.10.16.3-microsoft-standard-WSL2","mac":["02:42:ac:11:00:02"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.3 LTS (Focal Fossa)","major":20,"minor":4,"patch":3,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0,"id":"e4aa44123ef6412c8a2ef09168dec44b"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-11T01:58:52.664Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1093},"message":"Process info","service.name":"apm-server","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null},"cwd":"/","exe":"/usr/share/apm-server/bin/apm-server","name":"apm-server","pid":1159,"ppid":1158,"seccomp":{"mode":"filter","no_new_privs":false},"start_time":"2022-05-11T01:58:52.190Z"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-11T01:58:52.664Z","log.origin":{"file.name":"instance/beat.go","file.line":323},"message":"Setup Beat: apm-server; Version: 8.1.0","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-11T01:58:52.665Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":105},"message":"elasticsearch url: http://localhost:9200","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-11T01:58:52.665Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: c15db73e9a4a","service.name":"apm-server","ecs.version":"1.6.0"}
Config OK

Checking the server status it showed server was not available

Checking /var/log/apm-server I saw this error

{"log.level":"error","@timestamp":"2022-05-11T01:59:21.384Z","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":62},"message":"precondition 'apm integration installed' failed: error querying Elasticsearch for integration index templates: unexpected HTTP status: 404 Not Found ({\"error\":{\"root_cause\":[{\"type\":\"resource_not_found_exception\",\"reason\":\"index template matching [logs-apm.error] not found\"}],\"type\":\"resource_not_found_exception\",\"reason\":\"index template matching [logs-apm.error] not found\"},\"status\":404}): to remediate, please install the apm integration: https://ela.st/apm-integration-quickstart","service.name":"apm-server","ecs.version":"1.6.0"}

I didn't update apm-server.yml since I didn't know the what is elasticsearch's username/password - Is this the issue? If so is there any instruction on how to add an user for elasticsearch? Thank you

[spujadas]

Generally speaking I won’t be able to help you on how to set up APM, and would recommend heading over to the Elastic forums for guidance.

Having said that, going to the URL that’s referenced in the error in your APM logs (https://www.elastic.co/guide/en/apm/guide/current/apm-quick-start.html), there are several prerequisites on the set-up of the ELK stack, including having a "Secure, encrypted connection between Kibana and Elasticsearch."

As the documentation of the image says, the services are not set up with security enabled, so you’ll have to update the configuration of the services to do so (which can be a bit fiddly, hence this not being the case by default in the image).

Going back to the APM documentation, in the Prerequisites section, Self-managed tab, there are instructions to quickly secure Elasticsearch and Kibana for testing purposes, so that would be my recommendation to move forward with your set-up.

[duongpc]

Apparently ELK 8+ requires fleet setup to use APM, I was under assumption that I could get away from it by setting up APM legacy server, turns out fleet is mandatory...

I decided to downgrade to build 7.17.1

FROM sebp/elk:7.17.1

RUN curl -L -O https://artifacts.elastic.co/downloads/apm-server/apm-server-7.17.1-amd64.deb
RUN dpkg -i apm-server-7.17.1-amd64.deb

EXPOSE 8200

I was able to get kibana to recognize the APM server, however, from windows's command prompt, if I run:

curl localhost:8200
curl: (52) Empty reply from server

but inside the container, I got some result

# curl localhost:8200
{
  "build_date": "2022-02-23T22:31:58Z",
  "build_sha": "990c162bf1d5106be95d1430c6f3e98df865df7d",
  "publish_ready": true,
  "version": "7.17.1"
}

I can't quite figure out what missing, does the base image require any additional setup to allow access to port 8200?

[spujadas]

Binding the exposed port to a port on the host machine should do the trick (e.g. see https://github.com/spujadas/elk-docker/blob/master/docker-compose.yml for ports 5601, 9200, 5044).

[duongpc]

So by default apm-server.yml is using localhost:8200, changing it to either 0.0.0.0:8200 or :8200 will fix the connection issue