[docker] Error response from daemon: Could not find the file / in container <container_id>

Description

Upon using either Grafana or InfluxDBv2 in a configured stack, the Docker Engine throws an error:

Error response from daemon: Could not find the file / in container <container_id>

Upon forcing docker compose --project-directory=deploy up the Grafana container exits with the following logs:

komponist_grafana  | Getting secret GF_SECURITY_ADMIN_PASSWORD from /run/secrets/grafana_admin_password
komponist_grafana  | /run.sh: line 59: /run/secrets/grafana_admin_password: No such file or directory
komponist_grafana exited with code 1

Similar is the case for influxdbv2 service:

Attaching to 87ff105b8811_komponist_influxdbv2
87ff105b8811_komponist_influxdbv2  | DOCKER_INFLUXDB_INIT_PASSWORD_FILE defined, but file not existing, skipping.
87ff105b8811_komponist_influxdbv2  | DOCKER_INFLUXDB_INIT_USERNAME_FILE defined, but file not existing, skipping.

The offender is the user parameter set to ansible_user_id most likely 1000 in the templates and the final generated docker-compose.yml

Environment Info

Branch: main Docker Version: v24.0.0 Docker Compose Version: v2.18.1 Ansible Version: 2.15.0

Solution

Based on trial and errors, it turns out that the updated Docker Engine now looks for getent into the container as opposed to the host getent causing conflicts on which user/user ID to run the container with.

Upon inspecting the Dockerfile for Grafana, InfluxDBv2 both the images use grafana and influxdb as their respective users within the image.

Upon setting user: grafana and user: influxdb to the template files for the respective services, the stack works as before.

Tasks

[x] Investigate all users within each image of services in Komponist
[x] adapt the user parameter for the services that are not root
[x] perform checks on services whose users are by default to root

Results

Service	User / ID
`traefik`	`root`/ 0
`timescaledb`	`root` / 0
`questdb`	`questdb` / 10001
`eclipse-mosquitto`	`root` / 0
`nodered`	`node-red` / 1000
`grafana`	`grafana` / 1000
`influxdbv1`	`root` / 0
`influxdbv2`	`influxdb` / 1000

Update

a workaround is merged. Waiting for next Docker Engine release:

Moby Milestone 24.0.3

Update

Moby v24.0.3 Released

Grafana

For a standalone Grafana docker-compose.grafana.yml file generated by Komponist:

services:
  grafana:
    image: docker.io/grafana/grafana-oss:9.5.1
    container_name: komponist_grafana
    environment:
      - GF_SECURITY_ADMIN_USER__FILE=/run/secrets/grafana_admin_username
      - GF_SECURITY_ADMIN_PASSWORD__FILE=/run/secrets/grafana_admin_password
      - GF_SERVER_ROOT_URL=/grafana
      - GF_SERVER_SERVE_FROM_SUB_PATH=true
    secrets:
      - grafana_admin_username
      - grafana_admin_password
    logging:
      options:
        max-size: "5m"
    ports:
     - "3000:3000"
    security_opt:
      - "no-new-privileges:true"
    user: "1000"
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

secrets:
    grafana_admin_username:
      environment: GRAFANA_ADMIN_USERNAME
    grafana_admin_password:
      environment: GRAFANA_ADMIN_PASSWORD

will result in:

[+] Running 1/0
 ✔ Container komponist_grafana  Created                                                                                                                                                                     0.0s
Attaching to komponist_grafana
komponist_grafana  | Getting secret GF_SECURITY_ADMIN_PASSWORD from /run/secrets/grafana_admin_password
komponist_grafana  | /run.sh: line 59: /run/secrets/grafana_admin_password: Permission denied
komponist_grafana exited with code 1

Investigations

Removing `user` from compose file

Upon removing user: "1000" and letting the container run as USER for the Grafana setting causes the same Permissions error. A way around it is to add explicitly the UID/GID of the container's user (grafana)

Output

[+] Running 1/0
 ✔ Container komponist_grafana  Recreated                                                                                                                                                                   0.1s
Attaching to komponist_grafana
komponist_grafana  | Getting secret GF_SECURITY_ADMIN_PASSWORD from /run/secrets/grafana_admin_password
komponist_grafana  | /run.sh: line 59: /run/secrets/grafana_admin_password: Permission denied
komponist_grafana exited with code 1

Since the container already has grafana user with id 1000 the user has no access to the /run/secrets which is generally owned by root

Workaround

services:
  grafana:
    image: docker.io/grafana/grafana-oss:9.5.1
    container_name: komponist_grafana
    environment:
      - GF_SECURITY_ADMIN_USER__FILE=/run/secrets/grafana_admin_username
      - GF_SECURITY_ADMIN_PASSWORD__FILE=/run/secrets/grafana_admin_password
      - GF_SERVER_ROOT_URL=/grafana
      - GF_SERVER_SERVE_FROM_SUB_PATH=true
    secrets:
      - source: grafana_admin_username
        uid: "1000"
        gid: "1000"
      - source: grafana_admin_password
        uid: "1000"
        gid: "1000"
    logging:
      options:
        max-size: "5m"
    ports:
     - "3000:3000"
    security_opt:
      - "no-new-privileges:true"
#    user: "1000"
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

secrets:
    grafana_admin_username:
      environment: GRAFANA_ADMIN_USERNAME
    grafana_admin_password:
      environment: GRAFANA_ADMIN_PASSWORD

Upon checking the UID/GID permissions

docker compose exec grafana stat -c "%a %U %G" /run/secrets/grafana_admin_{username,password}
400 grafana root
400 grafana root

Adding Description to mounted secrets with `user`

Add world-readable permissions (0440) to the secrets as follows and user: 1000 set back again

services:
  grafana:
    image: docker.io/grafana/grafana-oss:9.5.1
    container_name: komponist_grafana
    environment:
      - GF_SECURITY_ADMIN_USER__FILE=/run/secrets/grafana_admin_username
      - GF_SECURITY_ADMIN_PASSWORD__FILE=/run/secrets/grafana_admin_password
      - GF_SERVER_ROOT_URL=/grafana
      - GF_SERVER_SERVE_FROM_SUB_PATH=true
    secrets:
      - source: grafana_admin_username
        mode: 0440
      - source: grafana_admin_password
        mode: 0440
    logging:
      options:
        max-size: "5m"
    ports:
     - "3000:3000"
    security_opt:
      - "no-new-privileges:true"
    user: "1000"
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

secrets:
    grafana_admin_username:
      environment: GRAFANA_ADMIN_USERNAME
    grafana_admin_password:
      environment: GRAFANA_ADMIN_PASSWORD

Upon checking the UID/GID permissions

docker compose exec grafana stat -c "%a %U %G" /run/secrets/grafana_admin_{username,password}
440 root root
440 root root

InfluxDBv2

For the following stand-alone docker-compose.influxdbv2.yml file:

services:
  influxdbv2:
    image: docker.io/influxdb:2.6-alpine
    container_name: komponist_influxdbv2
    environment:
      - DOCKER_INFLUXDB_INIT_MODE=setup
      - DOCKER_INFLUXDB_INIT_ORG=komponistorg
      - DOCKER_INFLUXDB_INIT_BUCKET=komponistdb
      - DOCKER_INFLUXDB_INIT_RETENTION=7d
      - DOCKER_INFLUXDB_INIT_USERNAME_FILE=/run/secrets/influxdbv2_admin_username
      - DOCKER_INFLUXDB_INIT_PASSWORD_FILE=/run/secrets/influxdbv2_admin_password
      - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN_FILE=/run/secrets/influxdbv2_admin_token
      - INFLUXD_HTTP_BIND_ADDRESS=0.0.0.0:8087 # bind on 8087 to avoid conflict with influxdbv1 in case both are required
    secrets:
      - influxdbv2_admin_username
      - influxdbv2_admin_password
      - influxdbv2_admin_token
    security_opt:
      - "no-new-privileges:true"
    user: "1000"
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

secrets:
  influxdbv2_admin_username:
    environment: INFLUXDBV2_ADMIN_USERNAME
  influxdbv2_admin_password:
    environment: INFLUXDBV2_ADMIN_PASSWORD
  influxdbv2_admin_token:
    environment: INFLUXDBV2_ADMIN_TOKEN

throws the following output:

[+] Running 2/2
 ✔ Network secretserror_default    Created                                                                                                                                                                  0.0s
 ✔ Container komponist_influxdbv2  Created                                                                                                                                                                  0.1s
Attaching to komponist_influxdbv2
komponist_influxdbv2  | cat: can't open '/run/secrets/influxdbv2_admin_password': Permission denied
komponist_influxdbv2  | DOCKER_INFLUXDB_INIT_PASSWORD_FILE defined, but file not existing, skipping.
komponist_influxdbv2  | cat: can't open '/run/secrets/influxdbv2_admin_username': Permission denied
komponist_influxdbv2  | DOCKER_INFLUXDB_INIT_USERNAME_FILE defined, but file not existing, skipping.
komponist_influxdbv2  | 2023-07-07T10:21:02.    error   missing parameter, cannot init InfluxDB {"system": "docker", "parameter": "DOCKER_INFLUXDB_INIT_USERNAME"}
komponist_influxdbv2  | 2023-07-07T10:21:02.    error   missing parameter, cannot init InfluxDB {"system": "docker", "parameter": "DOCKER_INFLUXDB_INIT_PASSWORD"}
komponist_influxdbv2 exited with code 1

Investigations

Removing `user` from Compose file

komponist_influxdbv2  | cat: can't open '/run/secrets/influxdbv2_admin_password': Permission denied
komponist_influxdbv2  | DOCKER_INFLUXDB_INIT_PASSWORD_FILE defined, but file not existing, skipping.
komponist_influxdbv2  | cat: can't open '/run/secrets/influxdbv2_admin_username': Permission denied
komponist_influxdbv2  | DOCKER_INFLUXDB_INIT_USERNAME_FILE defined, but file not existing, skipping.

But the container still runs without initialization

Workaround


services:
  influxdbv2:
    image: docker.io/influxdb:2.6-alpine
    container_name: komponist_influxdbv2
    environment:
      - DOCKER_INFLUXDB_INIT_MODE=setup
      - DOCKER_INFLUXDB_INIT_ORG=komponistorg
      - DOCKER_INFLUXDB_INIT_BUCKET=komponistdb
      - DOCKER_INFLUXDB_INIT_RETENTION=7d
      - DOCKER_INFLUXDB_INIT_USERNAME_FILE=/run/secrets/influxdbv2_admin_username
      - DOCKER_INFLUXDB_INIT_PASSWORD_FILE=/run/secrets/influxdbv2_admin_password
      - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN_FILE=/run/secrets/influxdbv2_admin_token
      - INFLUXD_HTTP_BIND_ADDRESS=0.0.0.0:8087 # bind on 8087 to avoid conflict with influxdbv1 in case both are required
    secrets:
      - source: influxdbv2_admin_username
        uid: "1000"
        gid: "1000"
#        mode: 0440
      - source: influxdbv2_admin_password
        uid: "1000"
        gid: "1000"
#        mode: 0440
      - source: influxdbv2_admin_token
        uid: "1000"
        gid: "1000"
#        mode: 0440
    security_opt:
      - "no-new-privileges:true"
#    user: "1000"
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

secrets:
  influxdbv2_admin_username:
    environment: INFLUXDBV2_ADMIN_USERNAME
  influxdbv2_admin_password:
    environment: INFLUXDBV2_ADMIN_PASSWORD
  influxdbv2_admin_token:
    environment: INFLUXDBV2_ADMIN_TOKEN

With `user` and secrets description

working configuration


services:
  influxdbv2:
    image: docker.io/influxdb:2.6-alpine
    container_name: komponist_influxdbv2
    environment:
      - DOCKER_INFLUXDB_INIT_MODE=setup
      - DOCKER_INFLUXDB_INIT_ORG=komponistorg
      - DOCKER_INFLUXDB_INIT_BUCKET=komponistdb
      - DOCKER_INFLUXDB_INIT_RETENTION=7d
      - DOCKER_INFLUXDB_INIT_USERNAME_FILE=/run/secrets/influxdbv2_admin_username
      - DOCKER_INFLUXDB_INIT_PASSWORD_FILE=/run/secrets/influxdbv2_admin_password
      - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN_FILE=/run/secrets/influxdbv2_admin_token
      - INFLUXD_HTTP_BIND_ADDRESS=0.0.0.0:8087 # bind on 8087 to avoid conflict with influxdbv1 in case both are required
    secrets:
      - source: influxdbv2_admin_username
        uid: "1000"
        gid: "1000"
        mode: 0440
      - source: influxdbv2_admin_password
        uid: "1000"
        gid: "1000"
        mode: 0440
      - source: influxdbv2_admin_token
        uid: "1000"
        gid: "1000"
        mode: 0440
    security_opt:
      - "no-new-privileges:true"
    ports:
      - "8087:8087"
    user: "1000"
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

secrets:
  influxdbv2_admin_username:
    environment: INFLUXDBV2_ADMIN_USERNAME
  influxdbv2_admin_password:
    environment: INFLUXDBV2_ADMIN_PASSWORD
  influxdbv2_admin_token:
    environment: INFLUXDBV2_ADMIN_TOKEN

TimescaleDB

Since the container runs as root there isn't a problem, however for compliance to a structure the following configuration file can be used


services:
  timescale:
    image: docker.io/timescale/timescaledb:latest-pg15
    container_name: komponist_timescaledb
    environment:
      - POSTGRES_USER_FILE=/run/secrets/timescale_admin_username
      - POSTGRES_PASSWORD_FILE=/run/secrets/timescale_admin_password
      - POSTGRES_DB_FILE=/run/secrets/timescale_database
    secrets:
      - source: timescale_admin_username
        mode: 0440
      - source: timescale_admin_password
        mode: 0440
      - source: timescale_database
        mode: 0440
    security_opt:
      - "no-new-privileges:true"
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

secrets:
  timescale_admin_username:
    environment: TIMESCALEDB_ADMIN_USERNAME
  timescale_admin_password:
    environment: TIMESCALEDB_ADMIN_PASSWORD
  timescale_database:
    environment: TIMESCALEDB_DATABASE

shantanoo-desai / komponist