[Bug]: External network assignment not working

zewas-digital commented 10 months ago

Description

I'm trying to set up an mailu instance on Coolify. Therefore I use the Docker Compose Deployment. In my configuration I am referring to an external network I created beforehand. But if I run start the Deployment fails with the error time="2023-11-13T08:05:07+01:00" level=warning msg="a network with name mailu_default exists but was not created by compose. Set 'external: true' to use an existing network" network mailu_default was found but has incorrect label com.docker.compose.network set to "", even tough I define the network as external.

Strangely the deployment works if I remove the network default from the front container, it's reference in other containers doesn't throw the error.

What could be the reason for it? Is the front container started before the network is referenced?

Minimal Reproduction (if possible, example repository)

Start with this compose file:

# This file is auto-generated by the Mailu configuration wizard.
# Please read the documentation before attempting any change.
# Generated for compose flavor

version: '2.2'

services:

  # External dependencies
  redis:
    image: redis:alpine
    restart: always
    volumes:
      - "/data/mailu/redis:/data"
    depends_on:
      - resolver
    dns:
      - 192.168.203.254

  # Core services
  front:
    image: ${DOCKER_ORG:-ghcr.io/mailu}/${DOCKER_PREFIX:-}nginx:${MAILU_VERSION:-2.0}
    restart: always
    environment:
      - SECRET_KEY
      - SUBNET
      - DOMAIN
      - HOSTNAMES
      - POSTMASTER
      - TLS_FLAVOR
      - AUTH_RATELIMIT_IP
      - AUTH_RATELIMIT_USER
      - DISABLE_STATISTICS
      - ADMIN
      - WEBMAIL
      - API
      - WEBDAV
      - ANTIVIRUS
      - SCAN_MACROS
      - MESSAGE_SIZE_LIMIT
      - MESSAGE_RATELIMIT
      - FETCHMAIL_ENABLED
      - FETCHMAIL_DELAY
      - RECIPIENT_DELIMITER
      - DMARC_RUA
      - DMARC_RUF
      - WELCOME
      - WELCOME_SUBJECT
      - WELCOME_BODY
      - WEB_ADMIN
      - SITENAME
      - WEBSITE
      - COMPOSE_PROJECT_NAME
      - CREDENTIAL_ROUNDS
      - REAL_IP_HEADER
      - REAL_IP_FROM
      - PROXY_PROTOCOL
      - LOG_LEVEL
      - TZ
      - DEFAULT_SPAM_THRESHOLD
      - LD_PRELOAD
    logging:
      driver: journald
      options:
        tag: mailu-front
    networks:
      - default
    volumes:
      - "/data/mailu/certs:/certs"
      - "/data/mailu/overrides/nginx:/overrides:ro"
    depends_on:
      - resolver
    dns:
      - 192.168.203.254
    labels:
      - "traefik.enable=true"

      - "traefik.http.routers.mailu-http.rule=Host(`mail.example.com`) && PathPrefix(`/.well-known/acme-challenge/`)"
      - "traefik.http.routers.mailu-http.entrypoints=http"
      - "traefik.http.routers.mailu-http.service=mailu-http"
      - "traefik.http.routers.mailu-http.priority=1001"
      - "traefik.http.services.mailu-http.loadbalancer.server.port=80"

      - "traefik.tcp.routers.mailu-https.rule=HostSNI(`mail.example.com`)"
      - "traefik.tcp.routers.mailu-https.entrypoints=https"
      - "traefik.tcp.routers.mailu-https.tls.passthrough=true"
      - "traefik.tcp.routers.mailu-https.service=mailu-https"
      - "traefik.tcp.services.mailu-https.loadbalancer.server.port=443"
      - "traefik.tcp.services.mailu-https.loadbalancer.proxyProtocol.version=2"

      - "traefik.tcp.routers.smtp.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.smtp.entrypoints=smtp"
      - "traefik.tcp.routers.smtp.service=smtp"
      - "traefik.tcp.services.smtp.loadbalancer.server.port=25"
      - "traefik.tcp.services.smtp.loadbalancer.proxyProtocol.version=2"

      - "traefik.tcp.routers.submission.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.submission.entrypoints=submission"
      - "traefik.tcp.routers.submission.service=submission"
      - "traefik.tcp.services.submission.loadbalancer.server.port=587"
      - "traefik.tcp.services.submission.loadbalancer.proxyProtocol.version=2"

      - "traefik.tcp.routers.submissions.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.submissions.entrypoints=submissions"
      - "traefik.tcp.routers.submissions.service=submissions"
      - "traefik.tcp.services.submissions.loadbalancer.server.port=465"
      - "traefik.tcp.services.submissions.loadbalancer.proxyProtocol.version=2"

      - "traefik.tcp.routers.imap.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.imap.entrypoints=imap"
      - "traefik.tcp.routers.imap.service=imap"
      - "traefik.tcp.services.imap.loadbalancer.server.port=143"
      - "traefik.tcp.services.imap.loadbalancer.proxyProtocol.version=2"

      - "traefik.tcp.routers.imaps.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.imaps.entrypoints=imaps"
      - "traefik.tcp.routers.imaps.service=imaps"
      - "traefik.tcp.services.imaps.loadbalancer.server.port=993"
      - "traefik.tcp.services.imaps.loadbalancer.proxyProtocol.version=2"

      - "traefik.tcp.routers.pop3.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.pop3.entrypoints=pop3"
      - "traefik.tcp.routers.pop3.service=pop3"
      - "traefik.tcp.services.pop3.loadbalancer.server.port=110"
      - "traefik.tcp.services.pop3.loadbalancer.proxyProtocol.version=2"

      - "traefik.tcp.routers.pop3s.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.pop3s.entrypoints=pop3s"
      - "traefik.tcp.routers.pop3s.service=pop3s"
      - "traefik.tcp.services.pop3s.loadbalancer.server.port=995"
      - "traefik.tcp.services.pop3s.loadbalancer.proxyProtocol.version=2"

      - "traefik.tcp.routers.sieve.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.sieve.entrypoints=sieve"
      - "traefik.tcp.routers.sieve.service=sieve"
      - "traefik.tcp.services.sieve.loadbalancer.server.port=4190"
      - "traefik.tcp.services.sieve.loadbalancer.proxyProtocol.version=2"

  resolver:
    image: ${DOCKER_ORG:-ghcr.io/mailu}/${DOCKER_PREFIX:-}unbound:${MAILU_VERSION:-2.0}
    environment:
      - SECRET_KEY
      - SUBNET
      - DOMAIN
      - HOSTNAMES
      - POSTMASTER
      - TLS_FLAVOR
      - AUTH_RATELIMIT_IP
      - AUTH_RATELIMIT_USER
      - DISABLE_STATISTICS
      - ADMIN
      - WEBMAIL
      - API
      - WEBDAV
      - ANTIVIRUS
      - SCAN_MACROS
      - MESSAGE_SIZE_LIMIT
      - MESSAGE_RATELIMIT
      - FETCHMAIL_ENABLED
      - FETCHMAIL_DELAY
      - RECIPIENT_DELIMITER
      - DMARC_RUA
      - DMARC_RUF
      - WELCOME
      - WELCOME_SUBJECT
      - WELCOME_BODY
      - WEB_ADMIN
      - SITENAME
      - WEBSITE
      - COMPOSE_PROJECT_NAME
      - CREDENTIAL_ROUNDS
      - REAL_IP_HEADER
      - REAL_IP_FROM
      - PROXY_PROTOCOL
      - LOG_LEVEL
      - TZ
      - DEFAULT_SPAM_THRESHOLD
      - LD_PRELOAD
    restart: always
    networks:
      default:
        ipv4_address: 192.168.203.254

  admin:
    image: ${DOCKER_ORG:-ghcr.io/mailu}/${DOCKER_PREFIX:-}admin:${MAILU_VERSION:-2.0}
    restart: always
    environment:
      - SECRET_KEY
      - SUBNET
      - DOMAIN
      - HOSTNAMES
      - POSTMASTER
      - TLS_FLAVOR
      - AUTH_RATELIMIT_IP
      - AUTH_RATELIMIT_USER
      - DISABLE_STATISTICS
      - ADMIN
      - WEBMAIL
      - API
      - WEBDAV
      - ANTIVIRUS
      - SCAN_MACROS
      - MESSAGE_SIZE_LIMIT
      - MESSAGE_RATELIMIT
      - FETCHMAIL_ENABLED
      - FETCHMAIL_DELAY
      - RECIPIENT_DELIMITER
      - DMARC_RUA
      - DMARC_RUF
      - WELCOME
      - WELCOME_SUBJECT
      - WELCOME_BODY
      - WEB_ADMIN
      - SITENAME
      - WEBSITE
      - COMPOSE_PROJECT_NAME
      - CREDENTIAL_ROUNDS
      - REAL_IP_HEADER
      - REAL_IP_FROM
      - PROXY_PROTOCOL
      - LOG_LEVEL
      - TZ
      - DEFAULT_SPAM_THRESHOLD
      - LD_PRELOAD
    logging:
      driver: journald
      options:
        tag: mailu-admin
    volumes:
      - "/data/mailu/data:/data"
      - "/data/mailu/dkim:/dkim"
    depends_on:
      - redis
      - resolver
    dns:
      - 192.168.203.254

  imap:
    image: ${DOCKER_ORG:-ghcr.io/mailu}/${DOCKER_PREFIX:-}dovecot:${MAILU_VERSION:-2.0}
    restart: always
    environment:
      - SECRET_KEY
      - SUBNET
      - DOMAIN
      - HOSTNAMES
      - POSTMASTER
      - TLS_FLAVOR
      - AUTH_RATELIMIT_IP
      - AUTH_RATELIMIT_USER
      - DISABLE_STATISTICS
      - ADMIN
      - WEBMAIL
      - API
      - WEBDAV
      - ANTIVIRUS
      - SCAN_MACROS
      - MESSAGE_SIZE_LIMIT
      - MESSAGE_RATELIMIT
      - FETCHMAIL_ENABLED
      - FETCHMAIL_DELAY
      - RECIPIENT_DELIMITER
      - DMARC_RUA
      - DMARC_RUF
      - WELCOME
      - WELCOME_SUBJECT
      - WELCOME_BODY
      - WEB_ADMIN
      - SITENAME
      - WEBSITE
      - COMPOSE_PROJECT_NAME
      - CREDENTIAL_ROUNDS
      - REAL_IP_HEADER
      - REAL_IP_FROM
      - PROXY_PROTOCOL
      - LOG_LEVEL
      - TZ
      - DEFAULT_SPAM_THRESHOLD
      - LD_PRELOAD
    logging:
      driver: journald
      options:
        tag: mailu-imap
    volumes:
      - "/data/mailu/mail:/mail"
      - "/data/mailu/overrides/dovecot:/overrides:ro"
    depends_on:
      - front
      - resolver
    dns:
      - 192.168.203.254

  smtp:
    image: ${DOCKER_ORG:-ghcr.io/mailu}/${DOCKER_PREFIX:-}postfix:${MAILU_VERSION:-2.0}
    restart: always
    environment:
      - SECRET_KEY
      - SUBNET
      - DOMAIN
      - HOSTNAMES
      - POSTMASTER
      - TLS_FLAVOR
      - AUTH_RATELIMIT_IP
      - AUTH_RATELIMIT_USER
      - DISABLE_STATISTICS
      - ADMIN
      - WEBMAIL
      - API
      - WEBDAV
      - ANTIVIRUS
      - SCAN_MACROS
      - MESSAGE_SIZE_LIMIT
      - MESSAGE_RATELIMIT
      - FETCHMAIL_ENABLED
      - FETCHMAIL_DELAY
      - RECIPIENT_DELIMITER
      - DMARC_RUA
      - DMARC_RUF
      - WELCOME
      - WELCOME_SUBJECT
      - WELCOME_BODY
      - WEB_ADMIN
      - SITENAME
      - WEBSITE
      - COMPOSE_PROJECT_NAME
      - CREDENTIAL_ROUNDS
      - REAL_IP_HEADER
      - REAL_IP_FROM
      - PROXY_PROTOCOL
      - LOG_LEVEL
      - TZ
      - DEFAULT_SPAM_THRESHOLD
      - LD_PRELOAD
    logging:
      driver: journald
      options:
        tag: mailu-smtp
    volumes:
      - "/data/mailu/mailqueue:/queue"
      - "/data/mailu/overrides/postfix:/overrides:ro"
    depends_on:
      - front
      - resolver
    dns:
      - 192.168.203.254

  oletools:
    image: ${DOCKER_ORG:-ghcr.io/mailu}/${DOCKER_PREFIX:-}oletools:${MAILU_VERSION:-2.0}
    hostname: oletools
    restart: always
    networks:
      - noinet
    depends_on:
      - resolver
    dns:
      - 192.168.203.254

  antispam:
    image: ${DOCKER_ORG:-ghcr.io/mailu}/${DOCKER_PREFIX:-}rspamd:${MAILU_VERSION:-2.0}
    hostname: antispam
    restart: always
    environment:
      - SECRET_KEY
      - SUBNET
      - DOMAIN
      - HOSTNAMES
      - POSTMASTER
      - TLS_FLAVOR
      - AUTH_RATELIMIT_IP
      - AUTH_RATELIMIT_USER
      - DISABLE_STATISTICS
      - ADMIN
      - WEBMAIL
      - API
      - WEBDAV
      - ANTIVIRUS
      - SCAN_MACROS
      - MESSAGE_SIZE_LIMIT
      - MESSAGE_RATELIMIT
      - FETCHMAIL_ENABLED
      - FETCHMAIL_DELAY
      - RECIPIENT_DELIMITER
      - DMARC_RUA
      - DMARC_RUF
      - WELCOME
      - WELCOME_SUBJECT
      - WELCOME_BODY
      - WEB_ADMIN
      - SITENAME
      - WEBSITE
      - COMPOSE_PROJECT_NAME
      - CREDENTIAL_ROUNDS
      - REAL_IP_HEADER
      - REAL_IP_FROM
      - PROXY_PROTOCOL
      - LOG_LEVEL
      - TZ
      - DEFAULT_SPAM_THRESHOLD
      - LD_PRELOAD
    logging:
      driver: journald
      options:
        tag: mailu-antispam
    networks:
      - default
      - noinet
    volumes:
      - "/data/mailu/filter:/var/lib/rspamd"
      - "/data/mailu/overrides/rspamd:/overrides:ro"
    depends_on:
      - front
      - redis
      - oletools
      - antivirus
      - resolver
    dns:
      - 192.168.203.254

  # Optional services
  antivirus:
    image: ${DOCKER_ORG:-ghcr.io/mailu}/${DOCKER_PREFIX:-}clamav:${MAILU_VERSION:-2.0}
    restart: always
    environment:
      - SECRET_KEY
      - SUBNET
      - DOMAIN
      - HOSTNAMES
      - POSTMASTER
      - TLS_FLAVOR
      - AUTH_RATELIMIT_IP
      - AUTH_RATELIMIT_USER
      - DISABLE_STATISTICS
      - ADMIN
      - WEBMAIL
      - API
      - WEBDAV
      - ANTIVIRUS
      - SCAN_MACROS
      - MESSAGE_SIZE_LIMIT
      - MESSAGE_RATELIMIT
      - FETCHMAIL_ENABLED
      - FETCHMAIL_DELAY
      - RECIPIENT_DELIMITER
      - DMARC_RUA
      - DMARC_RUF
      - WELCOME
      - WELCOME_SUBJECT
      - WELCOME_BODY
      - WEB_ADMIN
      - SITENAME
      - WEBSITE
      - COMPOSE_PROJECT_NAME
      - CREDENTIAL_ROUNDS
      - REAL_IP_HEADER
      - REAL_IP_FROM
      - PROXY_PROTOCOL
      - LOG_LEVEL
      - TZ
      - DEFAULT_SPAM_THRESHOLD
      - LD_PRELOAD
    volumes:
      - "/data/mailu/filter:/data"
    depends_on:
      - resolver
    dns:
      - 192.168.203.254

networks:
  default:
    name: mailu_default
    external: true
  noinet:
    driver: bridge
    internal: true

Environment Variables:

SECRET_KEY=AAAAAAAAAAAAAAAA
SUBNET=192.168.203.0/24
DOMAIN=example.com
HOSTNAMES=mail.example.com
POSTMASTER=admin
TLS_FLAVOR=letsencrypt
AUTH_RATELIMIT_IP=5/hour
AUTH_RATELIMIT_USER=50/day
DISABLE_STATISTICS=True
ADMIN=true
WEBROOT_REDIRECT=/sso/login
WEBMAIL=none
API=false
WEBDAV=none
ANTIVIRUS=clamav
SCAN_MACROS=true
MESSAGE_SIZE_LIMIT=50000000
MESSAGE_RATELIMIT=200/day
FETCHMAIL_ENABLED=False
FETCHMAIL_DELAY=600
RECIPIENT_DELIMITER=+
DMARC_RUA=admin
DMARC_RUF=admin
WELCOME=false
WELCOME_SUBJECT="Welcome to your new email account"
WELCOME_BODY="Welcome to your new email account, if you can read this, then it is configured properly!"
WEB_ADMIN=/admin
SITENAME="Mailu"
WEBSITE=https://mail.example.com
COMPOSE_PROJECT_NAME=mailu
CREDENTIAL_ROUNDS=12
REAL_IP_HEADER=X-Real-IP
REAL_IP_FROM=192.168.203.0/24
PROXY_PROTOCOL=mail
LOG_LEVEL=WARNING
TZ=Europe/Vienna
DEFAULT_SPAM_THRESHOLD=80
LD_PRELOAD=/usr/lib/libhardened_malloc.so

Then remove the network defaulton the front container and try again. It should work now.

Exception or Error

time="2023-11-13T08:05:07+01:00" level=warning msg="a network with name mailu_default exists but was not created by compose.\nSet 'external: true' to use an existing network" network mailu_default was found but has incorrect label com.docker.compose.network set to ""

Version

v4.0.0-beta.123

zewas-digital commented 10 months ago

Have found a workaround, if you add the front container to the end of the list of services it works. So I think it's really some kind of timing error.

zewas-digital commented 10 months ago

There also is an error with the DNS-config. My admin container fails to start because it gets a timeout from the DNS server. If I start the project with docker compose up it works fine.

peaklabs-dev commented 3 weeks ago

This is what I find the parser doing: Parsed should be like this (like it is defined):

networks:
  default:
    name: mailu_default
    external: true
  noinet:
    driver: bridge
    internal: true

This it what it parses:

networks:
  default: null
  noinet: null
  b40ssog84008gg4okosocw4k:
    name: b40ssog84008gg4okosocw4k
    external: true

--> This just used default settings but I think this should not be the case as both are different see above and also this cause the bug (at least I think this is the problem)

peaklabs-dev commented 1 week ago

@zewas-digital There was a new compose parser released recently with a lot of fixes. Please update to the latest version an try again. Let me know if it is fixed?

peaklabs-dev commented 1 day ago

There was a new parser release recently. This should fix the problem. Please upgrade to the latest version and if this problem occurs again, please re-open the issue.

coollabsio / coolify