search

Kong / docker-kong

:monkey: Docker distribution for Kong
https://docs.konghq.com/gateway/latest/install/docker/
Apache License 2.0
1.39k stars 832 forks source link

Permission error in portainer logs after starting #690

Open FuckingToasters opened 9 months ago

FuckingToasters commented 9 months ago

Hello, so the portianer logs show following:

Error: mkdir failed for '/var/run/kong/logs': Permission denied (code 13)
  Run with --v (verbose) or --vv (debug) for more details

I checkd if the folder exist which was not the case so i created the folder and added chmod permission 777 to it, sadly the issue keeps being the same. I also slightly modified the compose file because i dont want to have it exposed on public ips, instead it should only run internally and then i make use of nginx proxy manager to connect it to a domain in order to access it outside of the vps.

Here the docker-compose.yml file i use as a portainer stack:

version: '3.0'

x-kong-config:
  &kong-env
  KONG_DATABASE: ${KONG_DATABASE:-off}
  KONG_PG_DATABASE: ${KONG_PG_DATABASE:-kong}
  KONG_PG_HOST: db
  KONG_PG_USER: ${KONG_PG_USER:-kong}
  KONG_PG_PASSWORD_FILE: /run/secrets/kong_postgres_password

services:
  kong-migrations:
    image: "${KONG_DOCKER_TAG:-kong:latest}"
    command: kong migrations bootstrap
    profiles: [ "database" ]
    depends_on:
      - db
    environment:
      <<: *kong-env
    secrets:
      - kong_postgres_password
    networks:
      webproxy:
        ipv4_address: 172.26.0.18
    restart: on-failure

  kong-migrations-up:
    image: "${KONG_DOCKER_TAG:-kong:latest}"
    command: kong migrations up && kong migrations finish
    profiles: [ "database" ]
    depends_on:
      - db
    environment:
      <<: *kong-env
    secrets:
      - kong_postgres_password
    networks:
      webproxy:
        ipv4_address: 172.26.0.19
    restart: on-failure

  kong:
    image: "${KONG_DOCKER_TAG:-kong:latest}"
    user: "${KONG_USER:-kong}"
    environment:
      <<: *kong-env
      KONG_ADMIN_ACCESS_LOG: /dev/stdout
      KONG_ADMIN_ERROR_LOG: /dev/stderr
      KONG_PROXY_LISTEN: "${KONG_PROXY_LISTEN:-0.0.0.0:8000}"
      KONG_ADMIN_LISTEN: "${KONG_ADMIN_LISTEN:-0.0.0.0:8001}"
      KONG_ADMIN_GUI_LISTEN: "${KONG_ADMIN_GUI_LISTEN:-0.0.0.0:8002}"
      KONG_PROXY_ACCESS_LOG: /dev/stdout
      KONG_PROXY_ERROR_LOG: /dev/stderr
      KONG_PREFIX: ${KONG_PREFIX:-/var/run/kong}
      KONG_DECLARATIVE_CONFIG: "/opt/kong/kong.yaml"
    secrets:
      - kong_postgres_password
    networks:
      webproxy:
        ipv4_address: 172.26.0.17

    #ports:
      #- "172.26.0.17:8000:8000/tcp"
      #- "172.26.0.17:8443:8443/tcp"
      #- "127.0.0.18:8001:8001/tcp"
      #- "127.0.0.18:8444:8444/tcp"
      #- "127.0.0.18:8002:8002/tcp"
    healthcheck:
      test: [ "CMD", "kong", "health" ]
      interval: 10s
      timeout: 10s
      retries: 10
    restart: on-failure:5
    read_only: true
    volumes:
      - prefix_vol:${KONG_PREFIX:-/var/run/kong}
      - tmp_vol:/tmp
      - ./config:/opt/kong
    security_opt:
      - no-new-privileges

  db:
    image: postgres:latest
    profiles: [ "database" ]
    environment:
      POSTGRES_DB: ${KONG_PG_DATABASE:-kong}
      POSTGRES_USER: ${KONG_PG_USER:-kong}
      POSTGRES_PASSWORD_FILE: /run/secrets/kong_postgres_password
    secrets:
      - kong_postgres_password
    healthcheck:
      test:
        [
          "CMD",
          "pg_isready",
          "-d",
          "${KONG_PG_DATABASE:-kong}",
          "-U",
          "${KONG_PG_USER:-kong}"
        ]
      interval: 30s
      timeout: 30s
      retries: 3
    restart: on-failure
    stdin_open: true
    tty: true
    networks:
      webproxy:
        ipv4_address: 172.26.0.205
    volumes:
      - pgdata:/var/lib/postgresql/data

networks:
  webproxy:
    external: true

secrets:
  kong_postgres_password:
    file: /run/secrets/./kong_postgres_password

volumes:
  pgdata:
  prefix_vol:
  tmp_vol:
Tieske commented 9 months ago

@flrgh is this related to the prefix location stuff?

Tieske commented 9 months ago

@brentos fyi

flrgh commented 9 months ago

@flrgh is this related to the prefix location stuff?

This seems to me like a frustrating chicken/egg scenario that I've run into time and time again with docker.

Your prefix volume will be owned by root when docker[-compose] creates it and mounts it into the container at /var/run/kong, so it cannot be used by an unprivileged user without some initialization. You have a couple options:

use the default kong prefix of /usr/local/kong

The images are built with /usr/local/kong as the preferred prefix directory, which is created when the kong package is installed and then properly chown-ed to the kong user.

I have yet to find docker documentation that adequately explains this behavior, but when creating a container, mounted volumes interact with the image's filesystem layer in such a way that things are retained from the image layer and even promoted to the volume. Check this out:

t.sh ```shell #!/usr/bin/env bash export KONG_PREFIX=$1 export IMAGE=${2:-kong} volume=$(docker volume create) trap 'echo "removing volume $volume"; docker volume rm "$volume" > /dev/null' EXIT echo "volume $volume will be mounted at $KONG_PREFIX" echo "checking permissions of $KONG_PREFIX" docker run \ --rm \ -it \ -e "KONG_PREFIX=$KONG_PREFIX" \ --volume "$volume:$KONG_PREFIX" \ --entrypoint /bin/bash \ --read-only \ --user kong \ "$IMAGE" \ -c "ls -ld \"$KONG_PREFIX\"" echo "creating a file within $KONG_PREFIX" docker run \ --rm \ -it \ -e "KONG_PREFIX=$KONG_PREFIX" \ --volume "$volume:$KONG_PREFIX" \ --entrypoint /bin/bash \ --read-only \ --user kong \ "$IMAGE" \ -c "touch \"$KONG_PREFIX\"/test" host_dir=$(docker volume inspect "$volume" | jq -r '.[].Mountpoint') echo "view of $KONG_PREFIX from the host ($host_dir)" sudo ls -la "$host_dir" ```
./t.sh /var/run/kong ```console $ ./t.sh /var/run/kong volume 1e152c065634d3a8ecfee1b02eed3ffc3b8a30b39f8955d7ada52f2472603467 will be mounted at /var/run/kong checking permissions of /var/run/kong drwxr-xr-x. 2 root root 4096 Dec 13 19:08 /var/run/kong creating a file within /var/run/kong touch: cannot touch '/var/run/kong/test': Permission denied view of /var/run/kong from the host (/mnt/fast/docker/volumes/1e152c065634d3a8ecfee1b02eed3ffc3b8a30b39f8955d7ada52f2472603467/_data) total 8 drwxr-xr-x. 2 root root 4096 Dec 13 11:08 . drwx-----x. 3 root root 4096 Dec 13 11:08 .. removing volume 1e152c065634d3a8ecfee1b02eed3ffc3b8a30b39f8955d7ada52f2472603467 ```
./t.sh /some/other/path ```console $ ./t.sh /some/other/path volume 39614ff5db125822b05931c065b49beb37cf729a63438d29f4cc577da490db3a will be mounted at /some/other/path checking permissions of /some/other/path drwxr-xr-x. 2 root root 4096 Dec 13 19:12 /some/other/path creating a file within /some/other/path touch: cannot touch '/some/other/path/test': Permission denied view of /some/other/path from the host (/mnt/fast/docker/volumes/39614ff5db125822b05931c065b49beb37cf729a63438d29f4cc577da490db3a/_data) total 8 drwxr-xr-x. 2 root root 4096 Dec 13 11:12 . drwx-----x. 3 root root 4096 Dec 13 11:12 .. removing volume 39614ff5db125822b05931c065b49beb37cf729a63438d29f4cc577da490db3a ```
./t.sh /usr/local/kong ```console $ ./t.sh /usr/local/kong volume 5217cede029d0fb7b62cdfea66342943e56993809d3f285657c0b3a382b44f61 will be mounted at /usr/local/kong checking permissions of /usr/local/kong drwxrwxr-x. 5 kong root 4096 Dec 13 19:15 /usr/local/kong creating a file within /usr/local/kong view of /usr/local/kong from the host (/mnt/fast/docker/volumes/5217cede029d0fb7b62cdfea66342943e56993809d3f285657c0b3a382b44f61/_data) total 20 drwxrwxr-x. 5 michaelm root 4096 Dec 13 11:15 . drwx-----x. 3 root root 4096 Dec 13 11:15 .. drwxrwxr-x. 3 michaelm root 4096 Dec 13 11:15 gui drwxrwxr-x. 6 michaelm root 4096 Dec 13 11:15 include drwxrwxr-x. 5 michaelm root 4096 Dec 13 11:15 lib lrwxrwxrwx. 1 michaelm root 26 Nov 8 00:43 modules -> ../openresty/nginx/modules -rw-r--r--. 1 michaelm michaelm 0 Dec 13 11:15 test removing volume 5217cede029d0fb7b62cdfea66342943e56993809d3f285657c0b3a382b44f61 ``` Note: `michaelm` on my host system shares the same UID as `kong` within the container (1000), so that's why `ls` reports as such.

You can observe this behavior with (I think) any image and any arbitrary directory. Let's try with a directory /kong within an ubuntu:jammy-based image:

Dockerfile ```dockerfile FROM ubuntu:jammy USER root RUN useradd -u 1000 kong \ && mkdir -p /kong \ && chown kong:0 /kong \ && echo 'hello from my test image' > /kong/image ```
./t.sh /var/run/kong my-test-image ```console volume f2fdf4aa445b8279801b228ea1a462a969007aadaae7a7f04b460a19f08e7f5d will be mounted at /var/run/kong checking permissions of /var/run/kong drwxr-xr-x. 2 root root 4096 Dec 13 19:24 /var/run/kong creating a file within /var/run/kong touch: cannot touch '/var/run/kong/test': Permission denied view of /var/run/kong from the host (/mnt/fast/docker/volumes/f2fdf4aa445b8279801b228ea1a462a969007aadaae7a7f04b460a19f08e7f5d/_data) total 8 drwxr-xr-x. 2 root root 4096 Dec 13 11:24 . drwx-----x. 3 root root 4096 Dec 13 11:24 .. removing volume f2fdf4aa445b8279801b228ea1a462a969007aadaae7a7f04b460a19f08e7f5d ```
./t.sh /kong my-test-image ```console $ ./t.sh /kong test volume 815eb3cb52269372f32cb60fe6cf8a8de8ec1efd036e83aab903a34150424312 will be mounted at /kong checking permissions of /kong drwxr-xr-x. 2 kong root 4096 Dec 13 19:25 /kong creating a file within /kong view of /kong from the host (/mnt/fast/docker/volumes/815eb3cb52269372f32cb60fe6cf8a8de8ec1efd036e83aab903a34150424312/_data) total 12 drwxr-xr-x. 2 michaelm root 4096 Dec 13 11:25 . drwx-----x. 3 root root 4096 Dec 13 11:25 .. -rw-r--r--. 1 root root 25 Dec 13 11:25 image -rw-r--r--. 1 michaelm michaelm 0 Dec 13 11:25 test removing volume 815eb3cb52269372f32cb60fe6cf8a8de8ec1efd036e83aab903a34150424312 ```

use the init container pattern to initialize the prefix before kong starts

If you don't want to use /usr/local/kong for some reason, you'll need to do some more work to initialize your desired prefix directory before starting kong:

  kong-init:
    image: alpine:latest
    volumes:
      - prefix_vol:/kong-prefix
    # must use the uid/gid here because the user 'kong' won't
    # exist in your bare alpine image
    command: chown -Rv 1000:1000 /kong-prefix

...then add kong-init to depends_on for your kong service.