Persist login credentials in docker

[strangePo]

Hi everyone and sorry for this maybe simple question.

I'm running ntopng in a docker container using docker-compose. But the login credentials (changed after startup) aren't persisted between restarts.

This is my docker-compose file:

version: "3.5"
services:
  ntopng:
    restart: unless-stopped
    environment:
      TZ: "Europe/Berlin"
    image: "ntop/ntopng:stable"
    network_mode: host
    volumes:
      - ./data/ntopng:/var/lib/ntopng
    command: --community -d /var/lib/ntopng -i eth0 -w 0.0.0.0:3000

notpng version within the container:

v.4.2.201105    [Enterprise/Professional build]
GIT rev:        4.2-stable:97cccbb62db5b735a5a97cf6a3b76bc64d7bcbb4:20201105
Pro rev:        r3327
Built on:       Ubuntu 20.04.1 LTS
System Id:      6F19DB4C82056206
Platform:       x86_64
Edition:        Community
License Type:   Community
Maintenance:

I thought they will get saved within the data directory but that doesn't seem to be the case. Can someone tell me what i'm doing wrong?

Thank you

[simonemainardi]

By default, data in docker is not persistent. ntopng stores hashed passwords in Redis. Hence, you should persist Redis as explained here https://stackoverflow.com/questions/43341143/how-to-save-a-docker-redis-container/43341271#43341271

Alternatively, you can use option -r to instruct the containerized ntopng to connect to an external via network or socket file.

[strangePo]

Ok, thanks.

I already tried to persist the redis data by adding - ./data/redis:/var/lib/redis to the volumes section but the directory stayed empty. I'm not very much into redis so my knowledge about it is very little. I looked at the redis config file and it seems that it dumps the data in specified Intervalls (?). So i let it run for while and now the directory contains a dump.rdb file and the new credentials are working fine. The final docker-compose looks like this.

version: "3.5"
services:
  ntopng:
    restart: unless-stopped
    environment:
      TZ: "Europe/Berlin"
    image: "ntop/ntopng:stable"
    network_mode: host
    volumes:
      - ./data/ntopng:/var/lib/ntopng
      - ./data/redis:/var/lib/redis
    command: -d /var/lib/ntopng -i eth0 -w 0.0.0.0:3000

[cLysen]

Had the same issue, solved it by running a separate redis container. These are my docker run commands:

redis

docker run -d           \
--name ntopng-redis     \
-p 6379:6379            \
-v /docker_data/ntopng/redis_data:/data         \
redis redis-server --appendonly yes --requirepass <somepassword>

ntopng

docker run -d   \
--name ntopng   \
--net=host      \
-e TZ="Europe/Berlin"   \
-v /docker_data/ntopng/data:/var/lib/ntopng     \
ntop/ntopng:stable -i eth2 -w 0.0.0.0:3000 --redis localhost:6379:<password> --community

[fahrenhe1t]

Here's a working docker-compose.yml on default http port 3000:

version: '3.3'
services:
  redis:
    image: 'redis:latest'
    container_name: redis
    command: 'redis-server --appendonly yes --requirepass <redis_password>'
    restart: always
    ports:
      - '6379:6379'
    volumes:
      - 'redis:/data'
  ntopng:
    image: 'ntop/ntopng:latest'
    container_name: ntopng
    command: '--community -d /var/lib/ntopng -w 3000 -W 0 -i enp1s0 --redis localhost:6379:<redis_password>'
    restart: unless-stopped
    environment:
      TZ: 'Europe/Berlin'
    logging:
      options:
        max-size: 1g
    network_mode: host
    depends_on:
      - redis
    volumes:
      - '/var/run/docker.sock:/tmp/docker.sock:ro'
      - 'config:/var/lib/ntopng'
volumes:
  redis:
  config:

[williamjoy]

It's a good idea to trigger save before container shutdown, example in container entrypoint.sh

#!/bin/bash
setsid /usr/bin/redis-server /etc/redis/redis.conf &
trap "{ echo Received SIGTERM, saving redis data; redis-cli <<< save ; }" SIGTERM
trap "{ echo Received SIGINT, saving redis data; redis-cli <<< save; }" SIGINT
ntopng "$@" $NTOP_CONFIG

redis-cli <<< save sending save command before service stop

[soakes]

I am experencing the exact same issue and while I have tried various adjustments using the data within this post, it still refuses to save the user/creds or any extra settings you set within UI.

If I shutdown JUST the ntopng container, it will loose all settings. Any ideas what I am still doing wrong?

Many thanks.

version: '3'

networks:
  net:
    driver: bridge
  redis:
    driver: bridge

services:

  ntopng:
    image: ntop/ntopng:stable
    hostname: ntopng
    container_name: ntopng
    restart: unless-stopped
    networks:
      - net
      - redis
    ports:
      - 3000:3000
    #command: [ '-i', 'tcp://*:5556c', '-i', 'tcp://netflow2ng:5556', '-F', 'clickhouse;clickhouse;ntopng;clickhouse;default', '--disable-login', '-r', 'redis', '--community' ]
    command: [ '-i', 'tcp://*:5556c', '-i', 'tcp://netflow2ng:5556', '-F', 'clickhouse;clickhouse;ntopng;clickhouse;default', '--disable-login', '-r', 'redis', '--community', '-d', '/var/lib/ntopng', '-w', '3000', '-W', '0' ]
    volumes:
      - ./data_dir:/var/lib/ntopng
    logging:
      options:
        max-size: 1g
    cap_add:
      - NET_ADMIN
      - NET_RAW
    depends_on:
      - clickhouse

  netflow2ng:
    container_name: netflow2ng
    restart: unless-stopped
    image: synfinatic/netflow2ng:v0.0.3
    entrypoint: /netflow2ng # --level=debug
    networks:
      - net
    ports:
      - 5556:5556/tcp
      - 8080:8080/tcp
      - 2055:2055/udp
    depends_on:
      - ntopng

  clickhouse:
    image: clickhouse/clickhouse-server:latest
    hostname: clickhouse
    container_name: clickhouse
    restart: unless-stopped
    networks:
      - net
    volumes:
      - ./db_dir:/var/lib/clickhouse
      - ./logs_dir:/var/log/clickhouse-server
    environment:
      - CLICKHOUSE_DB=ntopng
      - CLICKHOUSE_USER=clickhouse
      - CLICKHOUSE_PASSWORD=default
      - CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1

  redis:
    image: redis:7.0.11-alpine
    container_name: redis
    command: redis-server --appendonly yes
    networks:
      - redis
    volumes:
      - ./redis_dir:/data
    restart: unless-stopped

[williamjoy]

I am experencing the exact same issue and while I have tried various adjustments using the data within this post, it still refuses to save the user/creds or any extra settings you set within UI.

If I shutdown JUST the ntopng container, it will loose all settings. Any ideas what I am still doing wrong?

Likely the redis data is not persisted onto disk. Can you try to run redis-cli, then save inside redis container before shutting down?

It worked for me with this redis customized container entrypoint, see https://github.com/ntop/ntopng/issues/4978#issuecomment-1426381168

[soakes]

I am experencing the exact same issue and while I have tried various adjustments using the data within this post, it still refuses to save the user/creds or any extra settings you set within UI. If I shutdown JUST the ntopng container, it will loose all settings. Any ideas what I am still doing wrong?

Likely the redis data is not persisted onto disk. Can you try to run redis-cli, then save inside redis container before shutting down?

It worked for me with this redis customized container entrypoint, see #4978 (comment)

Thanks very much for the info @williamjoy. I can confirm that as long as a redis-cli save command is issued inside the ntopng container before restart, does save all setting inside.

I will override the entrypoint as suggsed, thank you.

[soakes]

For completeness, I am enclosing my config for others and the reasoning behind the changes.

Iv'e managed to get the internal redis server slighly tweaked and figured the issue with the missing files. If you check inside the ntopng container, you will find the redis service is running as user redis with the uid of 102 and gid of 103. If you check outside the container, its been created by docker automaticly as root:root. Changing this to 102:103 resolves the saving of redis files.

I have updated the entrypoint script to make sure the files/dirs are set correctly, this saves any manual changes. If future containers are built with differnet uid/gid, ive set it to use the name, so it should keep working. I have also added a small tweak to redis which will save the db every minute as long as there has been one write.

I am also using netflow2ng image which is used to provide alternative method of importing in netflow data. I have also added an alias in the network section, so it shows a little nicer in the UI.

I have lastly added vlans which gets populated in the UI, to make it nicer and more feature ritch. You can call them what you like, but I prefer to have them listed with their numbers rather then description.

Thats about it, I hope this helps others with simular issues. Thanks again to @williamjoy for his method in fixing the "persistent data inc credentials not saved on restart"

docker-compose.yml

version: '3'

networks:
  net:
    driver: bridge
  db:
    driver: bridge

services:

  ntopng:
    image: ntop/ntopng:stable
    hostname: ntopng
    container_name: ntopng
    restart: unless-stopped
    volumes:
      - ./data_dir:/var/lib/ntopng
      - ./redis_dir:/var/lib/redis
      - ./entrypoint.sh:/entrypoint.sh
    cap_add:
      - NET_ADMIN
      - NET_RAW
    networks:
      - net
      - db
    ports:
      - 3000:3000
    entrypoint: ./entrypoint.sh # workaround by @williamjoy (https://github.com/ntop/ntopng/issues/4978#issuecomment-1905218166)
    command: [ '-i', 'tcp://*:5556c', '-i', 'tcp://router:5556', '-F', 'clickhouse;clickhouse;ntopng;ntopng;**********', '-m', '"10.29.10.0/24=vlan2910,10.29.20.0/24=vlan2920,10.29.30.0/24=vlan2930,10.29.50.0=vlan2950"', '--community' ]
    logging:
      options:
        max-size: 1g
    depends_on:
      - clickhouse

  netflow2ng:
    container_name: netflow2ng
    restart: unless-stopped
    image: synfinatic/netflow2ng:v0.0.3
    entrypoint: /netflow2ng # --level=debug
    networks:
      net:
        aliases:
          - router
    ports:
      - 5556:5556/tcp
      - 8080:8080/tcp
      - 2055:2055/udp
    depends_on:
      - ntopng

  clickhouse:
    image: clickhouse/clickhouse-server:latest
    hostname: clickhouse
    container_name: clickhouse
    restart: unless-stopped
    networks:
      - db
    volumes:
      - ./db_dir:/var/lib/clickhouse
      - ./logs_dir:/var/log/clickhouse-server
    environment:
      - CLICKHOUSE_DB=ntopng
      - CLICKHOUSE_USER=ntopng
      - CLICKHOUSE_PASSWORD=************
      - CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1

entrypoint.sh

#!/bin/bash
chown -R redis:redis /var/lib/redis
setsid redis-server /etc/redis/redis.conf --appendonly yes --save 60 1 &
trap "{ echo Received SIGTERM, saving redis data; redis-cli <<< save ; }" SIGTERM
trap "{ echo Received SIGINT, saving redis data; redis-cli <<< save; }" SIGINT
ntopng "$@" $NTOP_CONFIG

[nevusZ]

hy, what am i doing wrong here? root@docker:/doco/ntopng2# docker compose up [+] Running 1/0 ✔ Container clickhouse Running 0.0s Attaching to clickhouse, netflow2ng, ntopng2 Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "./entrypoint.sh": is a directory: unknown: permission denied

[soakes]

@nevusZ

It says why here :)

unable to start container process: exec: "./entrypoint.sh": is a directory: unknown: permission denied

You likley ran the docker compose before creating the file and so it created a directory instead. This is normal bahavior with docker if it doesnt exit.

Create the entrypoint.sh script first in the same dir as the docker-compose.yml file. If you have a directory called entrypoint.sh already there, remove it first then create script as shown above. Also make sure you chmod +x entrypoint.sh. After you done that, docker compose up -d should work as expected.

[nevusZ]

its working now, thank you forgot +x ;-)

[LearningToPi]

It's a good idea to trigger save before container shutdown, example in container entrypoint.sh

#!/bin/bash
setsid /usr/bin/redis-server /etc/redis/redis.conf &
trap "{ echo Received SIGTERM, saving redis data; redis-cli <<< save ; }" SIGTERM
trap "{ echo Received SIGINT, saving redis data; redis-cli <<< save; }" SIGINT
ntopng "$@" $NTOP_CONFIG

redis-cli <<< save sending save command before service stop

I took @williamjoy's script and tweaked it a bit. I created a 2nd script that runs in the background and saves the redis database every 5 minutes (you can of course change the time:

ntopng-save-redis.sh

#!/bin/bash
while true; do
    sleep 5m
    /usr/bin/redis-cli save > /dev/null 2>&1
done

Then reference is in my new entrypoint file: ntopng-entrypoint.sh

#!/bin/bash
/etc/init.d/redis-server start
trap "{ echo Received SIGTERM, saving redis data; redis-cli <<< save ; }" SIGTERM
trap "{ echo Received SIGINT, saving redis data; redis-cli <<< save; }" SIGINT
/ntopng-save-redis.sh &
ntopng "$@" $NTOP_CONFIG

Then you just need to map the scripts as well as the /var/lib/redis folder and set the entrypoint:

docker run -d --net host --name ntopng -v [..]/lib/ntop:/var/lib/ntop -v [..]/lib/redis:/var/lib/redis -v [..]/ntopng-entrypoint.sh:/ntopng-entrypoint.sh -v [..]/ntopng-save-redis.sh:/ntopng-save-redis.sh --entrypoint /ntopng-entrypoint.sh ntop/ntopng_arm64.dev:latest [...cmd parameters...]

Just make sure that the redis account has read/write access to the /var/lib/redis folder, and the ntop account read/write access to the /var/lib/ntop. Since it is a user inside the container, you can just grant the uid/gid access. If you are using docker namespaces make sure to add your starting subuid/subgid:

chown 999:999 [..]/lib/ntop
chown 106:110 [..]/lib/redis

(Alternatively you could use docker volumes and avoid the issues of access of files outside the container)