APM server does not start with default configuration

[dama-de]

Describe the bug While setting up elastdocker today, I noticed that the recently added APM server logs an error about the config file having the wrong permissions. I was able to fix this by running chmod g-w apm-server/data/apm-server.yml, this command could be included into the setup scripts. There is an alternative method via env var described in https://github.com/elastic/beats/issues/18858.

To Reproduce Spin up the default v3.2.0 elastdocker stack.

Expected behavior APM server starts without errors.

Screenshots

$ sudo docker compose up -d && sudo docker compose logs -f
[+] Running 6/6
 ⠿ Network elastic                      Created                                                                                0.1s
 ⠿ Volume "elastic_elasticsearch-data"  Created                                                                                0.0s
 ⠿ Container elastic-apm-server-1       Started                                                                                1.8s
 ⠿ Container elastic-elasticsearch-1    Started                                                                                1.2s
 ⠿ Container elastic-kibana-1           Started                                                                                1.6s
 ⠿ Container elastic-logstash-1         Started                                                                                1.9s
elastic-logstash-1       | Using bundled JDK: /usr/share/logstash/jdk
elastic-logstash-1       | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
elastic-apm-server-1     | Exiting: error loading config file: config file ("apm-server.yml") can only be writable by the owner but the permissions are "-rw-rw-r--" (to fix the permissions use: 'chmod go-w /usr/share/apm-server/apm-server.yml')
elastic-apm-server-1 exited with code 0
elastic-apm-server-1     | Exiting: error loading config file: config file ("apm-server.yml") can only be writable by the owner but the permissions are "-rw-rw-r--" (to fix the permissions use: 'chmod go-w /usr/share/apm-server/apm-server.yml')
elastic-apm-server-1     | Exiting: error loading config file: config file ("apm-server.yml") can only be writable by the owner but the permissions are "-rw-rw-r--" (to fix the permissions use: 'chmod go-w /usr/share/apm-server/apm-server.yml')
elastic-apm-server-1     | Exiting: error loading config file: config file ("apm-server.yml") can only be writable by the owner but the permissions are "-rw-rw-r--" (to fix the permissions use: 'chmod go-w /usr/share/apm-server/apm-server.yml')
elastic-apm-server-1 exited with code 1
[...]

Additional context I'm running on Ubuntu Server 20.04, Docker version 20.10.21, Docker Compose version 2.12.2.

[mostafahussein]

Hello @dama-de , Thanks for reporting, unfortunately, I was not able to reproduce it as per the attached logs. I tested it by cloning the repository and executing make setup then docker compose up -d

Is there anything missing in your steps that could help us reproduce the issue and identify the root cause?

[andliang]

I'm getting the same error. I also ran make setup prior to running docker compose up.

PS C:\Projects\elastdocker> docker compose up  
[+] Running 5/5
 - Network elastic                    Created                                                          0.0s 
 - Container elastic-apm-server-1     Created                                                          0.2s 
 - Container elastic-kibana-1         Created                                                          0.2s
 - Container elastic-elasticsearch-1  Created                                                          0.2s
 - Container elastic-logstash-1       Created                                                          0.2s
Attaching to elastic-apm-server-1, elastic-elasticsearch-1, elastic-kibana-1, elastic-logstash-1
elastic-logstash-1       | Using bundled JDK: /usr/share/logstash/jdk
elastic-logstash-1       | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
elastic-apm-server-1     | Exiting: error loading config file: config file ("apm-server.yml") can only be writable by the owner but the permissions are "-rwxrwxrwx" (to fix the permissions use: 'chmod go-w /usr/share/apm-server/apm-server.yml')
elastic-apm-server-1 exited with code 1
elastic-apm-server-1     | Exiting: error loading config file: config file ("apm-server.yml") can only be writable by the owner but the permissions are "-rwxrwxrwx" (to fix the permissions use: 'chmod go-w /usr/share/apm-server/apm-server.yml')
elastic-apm-server-1 exited with code 1
elastic-apm-server-1     | Exiting: error loading config file: config file ("apm-server.yml") can only be writable by the owner but the permissions are "-rwxrwxrwx" (to fix the permissions use: 'chmod go-w /usr/share/apm-server/apm-server.yml')
elastic-apm-server-1 exited with code 1
elastic-apm-server-1     | Exiting: error loading config file: config file ("apm-server.yml") can only be writable by the owner but the permissions are "-rwxrwxrwx" (to fix the permissions use: 'chmod go-w /usr/share/apm-server/apm-server.yml')
elastic-apm-server-1 exited with code 1
elastic-apm-server-1     | Exiting: error loading config file: config file ("apm-server.yml") can only be writable by the owner but the permissions are "-rwxrwxrwx" (to fix the permissions use: 'chmod go-w /usr/share/apm-server/apm-server.yml')
elastic-apm-server-1 exited with code 1
elastic-apm-server-1     | Exiting: error loading config file: config file ("apm-server.yml") can only be writable by the owner but the permissions are "-rwxrwxrwx" (to fix the permissions use: 'chmod go-w /usr/share/apm-server/apm-server.yml')
elastic-apm-server-1 exited with code 1
elastic-apm-server-1     | Exiting: error loading config file: config file ("apm-server.yml") can only be writable by the owner but the permissions are "-rwxrwxrwx" (to fix the permissions use: 'chmod go-w /usr/share/apm-server/apm-server.yml')
elastic-apm-server-1 exited with code 1

I'm running on Win 10, Docker version 20.10.21, build baeda1f, Docker version 20.10.21, build baeda1f

[mostafahussein]

Have you tried to fix the permission on the local config file as the error message suggests?

[0xMostafa]

@mostafahussein cannot update the permissions of local config

i tried

docker compose exec apm-server bash or docker compose exec apm-server ps aux

it responds with:

Error response from daemon: Container 7d427a3cdbc4c5efd69df1b770bfd3d790ba604b9c7d30c99c9bb2c006390c71 is restarting, wait until the container is running

the container status is Restarting (1) x seconds ago forever

[mostafahussein]

By the local config, I mean this file https://github.com/sherifabdlnaby/elastdocker/blob/main/apm-server/config/apm-server.yml, you don't need to enter the container to update the permission, just modify the original file and it will be mounted by docker compose, lets see if this will fix the issue or not

[0xMostafa]

Fixed

Thank you ya @mostafahussein

[andliang]

I tried modifying the original file by adding a benign white-space on windows, and that didn't work Needles to say, I can't modify the file permission by running 'chmod go-w /usr/share/apm-server/apm-server.yml' because that's not possible in windows.

So this is still an issue for me.

[andliang]

So I think found a workaround... I was able to resolve this error by adding command: --strict.perms=false for the apm-server service.

So for example:

apm-server:
    image: elastdocker/apm-server:${ELK_VERSION}
    build:
      context: apm-server/
      args:
        ELK_VERSION: $ELK_VERSION
    command: --strict.perms=false
    restart: unless-stopped
    ports:
      - "8200:8200"
    volumes:
      - ./apm-server/config/apm-server.yml:/usr/share/apm-server/apm-server.yml:r
    environment:
      ELASTIC_USERNAME: ${ELASTIC_USERNAME}
      ELASTIC_PASSWORD: ${ELASTIC_PASSWORD}
      ELASTICSEARCH_HOST_PORT: https://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}
      ELASTIC_APM_SECRET_TOKEN: ${ELASTIC_APM_SECRET_TOKEN}
    secrets:
      - source: elastic.ca
        target: /certs/ca.crt
      - source: apm-server.certificate
        target: /certs/apm-server.crt
      - source: apm-server.key
        target: /certs/apm-server.key

Read this issue for more info: Read this post for more info: https://github.com/elastic/apm-server/issues/3042