gravitl / netmaker

Netmaker makes networks with WireGuard. Netmaker automates fast, secure, and distributed virtual networks.
https://netmaker.io
Other
9.4k stars 547 forks source link

[Bug]: Docker mosquitto allways failes to start, certificates or openSSL issue? #1050

Closed fenpaws closed 2 years ago

fenpaws commented 2 years ago

Contact Details

soxxfenpa.ws

What happened?

I tried to set up two different Netmaker Servers with the Quick Install instructions. The first one that is official provided and a second one that replaces caddy with traefik.

With both, I get the same exact issue that mosquitto fails to start.

I redid the instructions multiple times, yet sill nothing.

Version

v0.13.0

What OS are you using?

Linux, Windows

Relevant log output

Attaching to mq
mq             | 1651221985: mosquitto version 2.0.14 starting
mq             | 1651221985: Config loaded from /mosquitto/config/mosquitto.conf.
mq             | 1651221985: Opening ipv4 listen socket on port 8883.
mq             | 1651221985: Opening ipv6 listen socket on port 8883.
mq             | 1651221985: Error: Unable to load CA certificates. Check cafile "/mosquitto/certs/root.pem".
mq             | 1651221985: Error: Unable to load server certificate "/mosquitto/certs/server.pem". Check certfile.
mq             | 1651221985: OpenSSL Error[0]: error:02FFF002:system library:func(4095):No such file or directory
mq             | 1651221985: OpenSSL Error[1]: error:20FFF080:BIO routines:CRYPTO_internal:no such file
mq             | 1651221985: OpenSSL Error[2]: error:0BFFF002:x509 certificate routines:CRYPTO_internal:system lib
mq             | 1651221985: OpenSSL Error[3]: error:02FFF002:system library:func(4095):No such file or directory
mq             | 1651221985: OpenSSL Error[4]: error:20FFF002:BIO routines:CRYPTO_internal:system lib
mq             | 1651221985: OpenSSL Error[5]: error:14FFF002:SSL routines:(UNKNOWN)SSL_internal:system lib

Contributing guidelines

mattkasun commented 2 years ago

certificates for mq were introduced in v0.13.0. It appears you are running version v0.12.2 of netmaker with a mosquitto.conf file from version v0.13.0

fenpaws commented 2 years ago

Ah yeah, i used the template that was provided to generate this bug report, there was no option for v0.13.0. I use the most recent version tho.

I changed mq to reflect the same version that was provided in quick start, same error tho

Here is my docker-compose.yml.

version: "3.4"

services:
  netmaker:
    container_name: netmaker
    image: gravitl/netmaker:v0.13.0
    volumes:
      - dnsconfig:/root/config/dnsconfig
      - sqldata:/root/data
    cap_add: 
      - NET_ADMIN
      - NET_RAW
      - SYS_MODULE
    sysctls:
      - net.ipv4.ip_forward=1
      - net.ipv4.conf.all.src_valid_mark=1
    restart: always
    environment:
      SERVER_HOST: "${NM_PUBLIC_IP}"
      SERVER_API_CONN_STRING: "api.${NM_BASE_DOMAIN}:443"
      SERVER_GRPC_CONN_STRING: "grpc.${NM_BASE_DOMAIN}:443"
      GRPC_SSL: "on"
      DNS_MODE: "on"
      SERVER_HTTP_HOST: "api.${NM_BASE_DOMAIN}"
      SERVER_GRPC_HOST: "grpc.${NM_BASE_DOMAIN}"
      API_PORT: "8081"
      GRPC_PORT: "50051"
      CLIENT_MODE: "on"
      MASTER_KEY: "${NM_MASTER_KEY}"
      CORS_ALLOWED_ORIGIN: "*"
      DISPLAY_KEYS: "on"
      DATABASE: "sqlite"
      NODE_ID: "netmaker-server-1"
      MQ_HOST: "mq"
      HOST_NETWORK: "off"
      MANAGE_IPTABLES: "on"
      PORT_FORWARD_SERVICES: "mq,dns"
      VERBOSITY: "1"
    ports:
      - "51821-51830:51821-51830/udp"
#      - "8081:8081"   # api endpoint exposed via traefik so not needed in ports
#      - "50051:50051" # grpc endpoint exposed via traefik so not needed in ports
  netmaker-ui:
    container_name: netmaker-ui
    depends_on:
      - netmaker
    image: gravitl/netmaker-ui:v0.13.0
    links:
      - "netmaker:api"
#    ports:
#      - "8082:80" # UI endpoint exposed via traefik so not needed in ports
    environment:
      BACKEND_URL: "https://api.${NM_BASE_DOMAIN}"
    restart: always
  coredns:
    depends_on:
      - netmaker
    image: coredns/coredns
    command: -conf /root/dnsconfig/Corefile
    container_name: coredns
    restart: always
    volumes:
      - dnsconfig:/root/dnsconfig
  mq:
    image: eclipse-mosquitto:2.0.11-openssl
    container_name: mq
    restart: unless-stopped
    ports:
      - "1883:1883"
    volumes:
      - ./data/mosquitto.conf:/mosquitto/config/mosquitto.conf
      - mosquitto_data:/mosquitto/data
      - mosquitto_logs:/mosquitto/log

volumes:
  sqldata:
    driver: local
    driver_opts:
      type: none
      device: /root/netmaker/data/netmaker_sqldata
      o: bind
  dnsconfig:
    driver: local
    driver_opts:
      type: none
      device: /root/netmaker/data/netmaker_dnsconfig
      o: bind
  mosquitto_data:
    driver: local
    driver_opts:
      type: none
      device: /root/netmaker/data/netmaker_mosquitto_data
      o: bind
  mosquitto_logs:
    driver: local
    driver_opts:
      type: none
      device: /root/netmaker/data/netmaker_mosquitto_logs
      o: bind
mattkasun commented 2 years ago

There are several issues with your compose file: bind mounts for /root/certs missing in netmaker /root/certs/:/etc/netmaker/ & mq /root/certs/:/etc/netmaker/

ssl port missing form mq /root/certs/:/etc/netmaker/

ip6 in netmaker

fenpaws commented 2 years ago

Okay, after these fixes it works, or at least I don't get an error any more. Thank you very much for the help. I post my current docker-compose.yml here so if someone finds it they can use it or compare.

version: "3.4"

services:
  netmaker:
    container_name: netmaker
    image: gravitl/netmaker:v0.13.0
    volumes:
      - ./data/dnsconfig:/root/config/dnsconfig
      - ./data/certs/:/etc/netmaker/
      - ./data/sqldata:/root/data
    cap_add: 
      - NET_ADMIN
      - NET_RAW
      - SYS_MODULE
    sysctls:
      - net.ipv4.ip_forward=1
      - net.ipv6.conf.all.disable_ipv6=0
      - net.ipv4.conf.all.src_valid_mark=1
    restart: always
    environment:
      SERVER_HOST: "${NM_PUBLIC_IP}"
      SERVER_API_CONN_STRING: "api.${NM_BASE_DOMAIN}:443"
      SERVER_GRPC_CONN_STRING: "grpc.${NM_BASE_DOMAIN}:443"
      GRPC_SSL: "on"
      DNS_MODE: "on"
      SERVER_HTTP_HOST: "api.${NM_BASE_DOMAIN}"
      SERVER_GRPC_HOST: "grpc.${NM_BASE_DOMAIN}"
      API_PORT: "8081"
      GRPC_PORT: "50051"
      CLIENT_MODE: "on"
      MASTER_KEY: "${NM_MASTER_KEY}"
      CORS_ALLOWED_ORIGIN: "*"
      DISPLAY_KEYS: "on"
      DATABASE: "sqlite"
      NODE_ID: "netmaker-server-1"
      MQ_HOST: "mq"
      HOST_NETWORK: "off"
      MANAGE_IPTABLES: "on"
      PORT_FORWARD_SERVICES: "mq,dns"
      VERBOSITY: "1"
    ports:
      - "51821-51830:51821-51830/udp"
#      - "8081:8081"   # api endpoint exposed via traefik so not needed in ports
#      - "50051:50051" # grpc endpoint exposed via traefik so not needed in ports
  netmaker-ui:
    container_name: netmaker-ui
    depends_on:
      - netmaker
    image: gravitl/netmaker-ui:v0.13.0
    links:
      - "netmaker:api"
#    ports:
#      - "8082:80" # UI endpoint exposed via traefik so not needed in ports
    environment:
      BACKEND_URL: "https://api.${NM_BASE_DOMAIN}"
    restart: always
  coredns:
    depends_on:
      - netmaker
    image: coredns/coredns
    command: -conf /root/dnsconfig/Corefile
    container_name: coredns
    restart: always
    volumes:
      - ./data/dnsconfig:/root/dnsconfig
  mq:
    image: eclipse-mosquitto:2.0.11-openssl
    container_name: mq
    restart: unless-stopped
    ports:
      - "1883:1883"
      - "8883:8883"
    volumes:
      - ./config/mosquitto.conf:/mosquitto/config/mosquitto.conf
      - ./data/mosquitto_data:/mosquitto/data
      - ./data/mosquitto_logs:/mosquitto/log
      - ./data/certs/:/mosquitto/certs/