Closed dl-lim closed 3 years ago
There is an issue template. Please use it!
Your issue is probably very easy to fix: don't use ultra slow disks. I see this on servers with slow spinning disks, where the io-wait basically explodes when you start multiple containers.
Or do as your log suggests: Increase the timeout of docker-compose.
Am 06.09.2020 um 03:55 schrieb alderson59 notifications@github.com:
I run into this same error several times when creating php-fpm and mysql containers.
On my qemu VM machine, I have bind-mounted the /srv/ directory onto a virtio 9p drive - refer to docker compose file. I've basically put every volume on this mounted device.
It works each time I do the docker-compose up -d with no pre-built files (i.e. I deleted it and it looks like a fresh install) However, I am testing it for stability and tried to do a docker-compose down, then docker-compose up again, but it always fails on the second up.
NOTE: I should note that when it works (i.e. the first time after a fresh install), I manage to get the whole system running - able to send and receive emails as expected. This issue is about testing the system for stability when I do a docker-compose down then up.
It always stops at php-fpm and mysql, with a timeout. I've tested setting timeout to 5mins, still no difference.
I suspect it could be due to the virtio drive - or maybe it isn't. And occassionally I see that mysql does not run when mysqld volume is on the mounted drive. So, I set the mysqld volume onto /srvlocal/ which is not on a mounted drive. Sometimes it works and sometimes it doesn't. Doing a docker-compose down then up doesn't always work as well as I'd hoped. There are too many variables here than I'm familiar with, and I'm still running into the same errors.
So my question is, which volumes are well-known to not work on a non-local drive?
docker-compose up: (sometimes, mysql works and postfix doesn't, but php-fpm always seems to be the culprit)
root@ubuntu-mail:/opt/mailcow-dockerized# docker-compose up -d Creating network "mailcow_mailcow-network" with driver "bridge" Creating mailcow_solr-mailcow_1 ... done Creating mailcow_sogo-mailcow_1 ... done Creating mailcow_dockerapi-mailcow_1 ... done Creating mailcow_clamd-mailcow_1 ... done Creating mailcow_olefy-mailcow_1 ... done Creating mailcow_redis-mailcow_1 ... done Creating mailcow_memcached-mailcow_1 ... done Creating mailcow_unbound-mailcow_1 ... done Creating mailcow_watchdog-mailcow_1 ... done Creating mailcow_php-fpm-mailcow_1 ... Creating mailcow_mysql-mailcow_1 ...
ERROR: for mailcow_php-fpm-mailcow_1 UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
ERROR: for mailcow_mysql-mailcow_1 UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
ERROR: for php-fpm-mailcow UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
ERROR: for mysql-mailcow UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60) ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information. If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60). root@ubuntu-mail:/opt/mailcow-dockerized# Also, once the above error shows, docker-compose down wouldn't work, and I cannot restart the docker service entirely. Even rebooting will timeout. I have to force stop the VM each time to recover.
mailcow.conf: [Summary: Only changed hostname, tz, watchdog email, compose name]
------------------------------
mailcow web ui configuration
------------------------------
example.org is not a valid hostname, use a fqdn here.
Default admin user is "admin"
Default password is "moohoo"
MAILCOW_HOSTNAME=xxxhostname
------------------------------
SQL database configuration
------------------------------
DBNAME=mailcow DBUSER=mailcow
Please use long, random alphanumeric strings (A-Za-z0-9)
DBPASS=xxxpass DBROOT=xxxroot
------------------------------
HTTP/S Bindings
------------------------------
You should use HTTPS, but in case of SSL offloaded reverse proxies:
Might be important: This will also change the binding within the container.
If you use a proxy within Docker, point it to the ports you set below.
Do not use IP:PORT in HTTP(S)_BIND or HTTP(S)_PORT
IMPORTANT: Do not use port 8081, 9081 or 65510!
HTTP_PORT=80 HTTP_BIND=0.0.0.0
HTTPS_PORT=443 HTTPS_BIND=0.0.0.0
------------------------------
Other bindings
------------------------------
You should leave that alone
Format: 11.22.33.44:25 or 0.0.0.0:465 etc.
SMTP_PORT=25 SMTPS_PORT=465 SUBMISSION_PORT=587 IMAP_PORT=143 IMAPS_PORT=993 POP_PORT=110 POPS_PORT=995 SIEVE_PORT=4190 DOVEADM_PORT=127.0.0.1:19991 SQL_PORT=127.0.0.1:13306 SOLR_PORT=127.0.0.1:18983 REDIS_PORT=127.0.0.1:7654
Your timezone
TZ=xxxsomewhere
Fixed project name
Please use lowercase letters only
COMPOSE_PROJECT_NAME=mailcow
Set this to "allow" to enable the anyone pseudo user. Disabled by default.
When enabled, ACL can be created, that apply to "All authenticated users"
This should probably only be activated on mail hosts, that are used exclusivly by one organisation.
Otherwise a user might share data with too many other users.
ACL_ANYONE=disallow
Garbage collector cleanup
Deleted domains and mailboxes are moved to /var/vmail/_garbage/timestamp_sanitizedstring
How long should objects remain in the garbage until they are being deleted? (value in minutes)
Check interval is hourly
MAILDIR_GC_TIME=1440
Additional SAN for the certificate
#
You can use wildcard records to create specific names for every domain you add to mailcow.
Example: Add domains "example.com" and "example.net" to mailcow, change ADDITIONAL_SAN to a value like:
ADDITIONAL_SAN=imap.,smtp.
This will expand the certificate to "imap.example.com", "smtp.example.com", "imap.example.net", "imap.example.net"
plus every domain you add in the future.
#
You can also just add static names...
ADDITIONAL_SAN=srv1.example.net
...or combine wildcard and static names:
ADDITIONAL_SAN=imap.*,srv1.example.com
#
ADDITIONAL_SAN=
Skip running ACME (acme-mailcow, Let's Encrypt certs) - y/n
SKIP_LETS_ENCRYPT=n
Create seperate certificates for all domains - y/n
this will allow adding more than 100 domains, but some email clients will not be able to connect with alternative hostnames
see https://wiki.dovecot.org/SSL/SNIClientSupport
ENABLE_SSL_SNI=n
Skip IPv4 check in ACME container - y/n
SKIP_IP_CHECK=n
Skip HTTP verification in ACME container - y/n
SKIP_HTTP_VERIFICATION=n
Skip ClamAV (clamd-mailcow) anti-virus (Rspamd will auto-detect a missing ClamAV container) - y/n
SKIP_CLAMD=n
Skip SOGo: Will disable SOGo integration and therefore webmail, DAV protocols and ActiveSync support (experimental, unsupported, not fully implemented) - y/n
SKIP_SOGO=n
Skip Solr on low-memory systems or if you do not want to store a readable index of your mails in solr-vol-1.
SKIP_SOLR=n
Solr heap size in MB, there is no recommendation, please see Solr docs.
Solr is a prone to run OOM and should be monitored. Unmonitored Solr setups are not recommended.
SOLR_HEAP=1024
Enable watchdog (watchdog-mailcow) to restart unhealthy containers
USE_WATCHDOG=y
Allow admins to log into SOGo as email user (without any password)
ALLOW_ADMIN_EMAIL_LOGIN=n
Send notifications by mail (sent from watchdog@MAILCOW_HOSTNAME)
CAUTION:
1. You should use external recipients
2. Mails are sent unsigned (no DKIM)
3. If you use DMARC, create a separate DMARC policy ("v=DMARC1; p=none;" in _dmarc.MAILCOW_HOSTNAME)
Multiple rcpts allowed, NO quotation marks, NO spaces
WATCHDOG_NOTIFY_EMAIL=a@example.com,b@example.com,c@example.com
WATCHDOG_NOTIFY_EMAIL=xxxxx@example.com
Notify about banned IP (includes whois lookup)
WATCHDOG_NOTIFY_BAN=y
Checks if mailcow is an open relay. Requires a SAL. More checks will follow.
https://www.servercow.de/mailcow?lang=en
https://www.servercow.de/mailcow?lang=de
No data is collected. Opt-in and anonymous.
Will only work with unmodified mailcow setups.
WATCHDOG_EXTERNAL_CHECKS=n
Max log lines per service to keep in Redis logs
LOG_LINES=9999
Internal IPv4 /24 subnet, format n.n.n (expands to n.n.n.0/24)
Use private IPv4 addresses only, see https://en.wikipedia.org/wiki/Private_network#Private_IPv4_addresses
IPV4_NETWORK=172.22.1
Internal IPv6 subnet in fc00::/7
Use private IPv6 addresses only, see https://en.wikipedia.org/wiki/Private_network#Private_IPv6_addresses
IPV6_NETWORK=fd4d:6169:6c63:6f77::/64
Use this IPv4 for outgoing connections (SNAT)
SNAT_TO_SOURCE=
Use this IPv6 for outgoing connections (SNAT)
SNAT6_TO_SOURCE=
Create or override an API key for the web UI
You must define API_ALLOW_FROM, which is a comma separated list of IPs
An API key defined as API_KEY has read-write access
An API key defined as API_KEY_READ_ONLY has read-only access
Allowed chars for API_KEY and API_KEY_READ_ONLY: a-z, A-Z, 0-9, -
You can define API_KEY and/or API_KEY_READ_ONLY
API_KEY=
API_KEY_READ_ONLY=
API_ALLOW_FROM=172.22.1.1,127.0.0.1
mail_home is ~/Maildir
MAILDIR_SUB=Maildir
SOGo session timeout in minutes
SOGO_EXPIRE_SESSION=480 docker-compose.yml: [Summary: Changed volume locations only - removed named volumes to save space on qemu image. Mounted directly on virtio drive, which is the same physical drive the image sits on]
version: '2.1' services:
unbound-mailcow: image: mailcow/unbound:1.12 environment: - TZ=${TZ} volumes: - ./data/hooks/unbound:/hooks - ./data/conf/unbound/unbound.conf:/etc/unbound/unbound.conf:ro restart: unless-stopped tty: true networks: mailcow-network: ipv4_address: ${IPV4_NETWORK:-172.22.1}.254 aliases: - unbound mysql-mailcow: image: mariadb:10.4 depends_on: - unbound-mailcow stop_grace_period: 45s volumes: - /srv/mailcow/mysql:/var/lib/mysql/ - /srvlocal/mailcow/mysqld:/var/run/mysqld/ - ./data/conf/mysql/:/etc/mysql/conf.d/:ro environment: - TZ=${TZ} - MYSQL_ROOT_PASSWORD=${DBROOT} - MYSQL_DATABASE=${DBNAME} - MYSQL_USER=${DBUSER} - MYSQL_PASSWORD=${DBPASS} - MYSQL_INITDB_SKIP_TZINFO=1 restart: unless-stopped ports: - "${SQL_PORT:-127.0.0.1:13306}:3306" networks: mailcow-network: aliases: - mysql redis-mailcow: image: redis:5-alpine volumes: - /srv/mailcow/redis/data:/data/ restart: unless-stopped ports: - "${REDIS_PORT:-127.0.0.1:7654}:6379" environment: - TZ=${TZ} networks: mailcow-network: ipv4_address: ${IPV4_NETWORK:-172.22.1}.249 aliases: - redis clamd-mailcow: image: mailcow/clamd:1.36 restart: unless-stopped dns: - ${IPV4_NETWORK:-172.22.1}.254 environment: - TZ=${TZ} - SKIP_CLAMD=${SKIP_CLAMD:-n} volumes: - ./data/conf/clamav/:/etc/clamav/ networks: mailcow-network: aliases: - clamd rspamd-mailcow: image: mailcow/rspamd:1.68 stop_grace_period: 30s depends_on: - dovecot-mailcow environment: - TZ=${TZ} - IPV4_NETWORK=${IPV4_NETWORK:-172.22.1} - IPV6_NETWORK=${IPV6_NETWORK:-fd4d:6169:6c63:6f77::/64} - REDIS_SLAVEOF_IP=${REDIS_SLAVEOF_IP:-} - REDIS_SLAVEOF_PORT=${REDIS_SLAVEOF_PORT:-} volumes: - ./data/hooks/rspamd:/hooks - ./data/conf/rspamd/custom/:/etc/rspamd/custom - ./data/conf/rspamd/override.d/:/etc/rspamd/override.d - ./data/conf/rspamd/local.d/:/etc/rspamd/local.d - ./data/conf/rspamd/plugins.d/:/etc/rspamd/plugins.d - ./data/conf/rspamd/lua/:/etc/rspamd/lua/:ro - ./data/conf/rspamd/rspamd.conf.local:/etc/rspamd/rspamd.conf.local - ./data/conf/rspamd/rspamd.conf.override:/etc/rspamd/rspamd.conf.override - /srv/mailcow/rspamd:/var/lib/rspamd restart: unless-stopped dns: - ${IPV4_NETWORK:-172.22.1}.254 hostname: rspamd networks: mailcow-network: aliases: - rspamd php-fpm-mailcow: image: mailcow/phpfpm:1.69 command: "php-fpm -d date.timezone=${TZ} -d expose_php=0" depends_on: - redis-mailcow volumes: - ./data/hooks/phpfpm:/hooks - ./data/web:/web:rw - ./data/conf/rspamd/dynmaps:/dynmaps:ro - ./data/conf/rspamd/custom/:/rspamd_custom_maps - /srv/mailcow/rspamd:/var/lib/rspamd - /srvlocal/mailcow/mysqld:/var/run/mysqld/ - ./data/conf/sogo/:/etc/sogo/ - ./data/conf/rspamd/meta_exporter:/meta_exporter:ro - ./data/conf/phpfpm/sogo-sso/:/etc/sogo-sso/ - ./data/conf/phpfpm/php-fpm.d/pools.conf:/usr/local/etc/php-fpm.d/z-pools.conf - ./data/conf/phpfpm/php-conf.d/opcache-recommended.ini:/usr/local/etc/php/conf.d/opcache-recommended.ini - ./data/conf/phpfpm/php-conf.d/upload.ini:/usr/local/etc/php/conf.d/upload.ini - ./data/conf/phpfpm/php-conf.d/other.ini:/usr/local/etc/php/conf.d/zzz-other.ini - ./data/conf/dovecot/global_sieve_before:/global_sieve/before - ./data/conf/dovecot/global_sieve_after:/global_sieve/after - ./data/assets/templates:/tpls dns: - ${IPV4_NETWORK:-172.22.1}.254 environment: - REDIS_SLAVEOF_IP=${REDIS_SLAVEOF_IP:-} - REDIS_SLAVEOF_PORT=${REDIS_SLAVEOF_PORT:-} - LOG_LINES=${LOG_LINES:-9999} - TZ=${TZ} - DBNAME=${DBNAME} - DBUSER=${DBUSER} - DBPASS=${DBPASS} - MAILCOW_HOSTNAME=${MAILCOW_HOSTNAME} - IMAP_PORT=${IMAP_PORT:-143} - IMAPS_PORT=${IMAPS_PORT:-993} - POP_PORT=${POP_PORT:-110} - POPS_PORT=${POPS_PORT:-995} - SIEVE_PORT=${SIEVE_PORT:-4190} - IPV4_NETWORK=${IPV4_NETWORK:-172.22.1} - IPV6_NETWORK=${IPV6_NETWORK:-fd4d:6169:6c63:6f77::/64} - SUBMISSION_PORT=${SUBMISSION_PORT:-587} - SMTPS_PORT=${SMTPS_PORT:-465} - SMTP_PORT=${SMTP_PORT:-25} - API_KEY=${API_KEY:-invalid} - API_KEY_READ_ONLY=${API_KEY_READ_ONLY:-invalid} - API_ALLOW_FROM=${API_ALLOW_FROM:-invalid} - COMPOSE_PROJECT_NAME=${COMPOSE_PROJECT_NAME:-mailcow-dockerized} - SKIP_SOLR=${SKIP_SOLR:-y} - SKIP_CLAMD=${SKIP_CLAMD:-n} - SKIP_SOGO=${SKIP_SOGO:-n} - ALLOW_ADMIN_EMAIL_LOGIN=${ALLOW_ADMIN_EMAIL_LOGIN:-n} - MASTER=${MASTER:-y} restart: unless-stopped networks: mailcow-network: aliases: - phpfpm sogo-mailcow: image: mailcow/sogo:1.83 environment: - DBNAME=${DBNAME} - DBUSER=${DBUSER} - DBPASS=${DBPASS} - TZ=${TZ} - LOG_LINES=${LOG_LINES:-9999} - MAILCOW_HOSTNAME=${MAILCOW_HOSTNAME} - ACL_ANYONE=${ACL_ANYONE:-disallow} - ALLOW_ADMIN_EMAIL_LOGIN=${ALLOW_ADMIN_EMAIL_LOGIN:-n} - IPV4_NETWORK=${IPV4_NETWORK:-172.22.1} - SOGO_EXPIRE_SESSION=${SOGO_EXPIRE_SESSION:-480} - SKIP_SOGO=${SKIP_SOGO:-n} - MASTER=${MASTER:-y} - REDIS_SLAVEOF_IP=${REDIS_SLAVEOF_IP:-} - REDIS_SLAVEOF_PORT=${REDIS_SLAVEOF_PORT:-} dns: - ${IPV4_NETWORK:-172.22.1}.254 volumes: - ./data/conf/sogo/:/etc/sogo/ - ./data/web/inc/init_db.inc.php:/init_db.inc.php - ./data/conf/sogo/custom-sogo.js:/usr/lib/GNUstep/SOGo/WebServerResources/js/custom-sogo.js - /srvlocal/mailcow/mysqld:/var/run/mysqld/ - /srv/mailcow/sogo_web:/sogo_web - /srv/mailcow/sogo_backup:/sogo_backup restart: unless-stopped networks: mailcow-network: ipv4_address: ${IPV4_NETWORK:-172.22.1}.248 aliases: - sogo dovecot-mailcow: image: mailcow/dovecot:1.130 depends_on: - mysql-mailcow dns: - ${IPV4_NETWORK:-172.22.1}.254 cap_add: - NET_BIND_SERVICE volumes: - ./data/hooks/dovecot:/hooks - ./data/conf/dovecot:/etc/dovecot - ./data/assets/ssl:/etc/ssl/mail/:ro - ./data/conf/sogo/:/etc/sogo/ - ./data/conf/phpfpm/sogo-sso/:/etc/phpfpm/ - /srv/mailcow/vmail:/var/vmail - /srv/mailcow/attachments:/var/attachments - /srv/mailcow/crypt:/mail_crypt/ - ./data/conf/rspamd/custom/:/etc/rspamd/custom - ./data/assets/templates:/templates - /srv/mailcow/rspamd:/var/lib/rspamd - /srvlocal/mailcow/mysqld:/var/run/mysqld/ environment: - LOG_LINES=${LOG_LINES:-9999} - DBNAME=${DBNAME} - DBUSER=${DBUSER} - DBPASS=${DBPASS} - TZ=${TZ} - MAILCOW_HOSTNAME=${MAILCOW_HOSTNAME} - IPV4_NETWORK=${IPV4_NETWORK:-172.22.1} - ALLOW_ADMIN_EMAIL_LOGIN=${ALLOW_ADMIN_EMAIL_LOGIN:-n} - MAILDIR_GC_TIME=${MAILDIR_GC_TIME:-1440} - ACL_ANYONE=${ACL_ANYONE:-disallow} - SKIP_SOLR=${SKIP_SOLR:-y} - MAILDIR_SUB=${MAILDIR_SUB:-} - MASTER=${MASTER:-y} - REDIS_SLAVEOF_IP=${REDIS_SLAVEOF_IP:-} - REDIS_SLAVEOF_PORT=${REDIS_SLAVEOF_PORT:-} - COMPOSE_PROJECT_NAME=${COMPOSE_PROJECT_NAME:-mailcow-dockerized} ports: - "${DOVEADM_PORT:-127.0.0.1:19991}:12345" - "${IMAP_PORT:-143}:143" - "${IMAPS_PORT:-993}:993" - "${POP_PORT:-110}:110" - "${POPS_PORT:-995}:995" - "${SIEVE_PORT:-4190}:4190" restart: unless-stopped tty: true ulimits: nproc: 65535 nofile: soft: 20000 hard: 40000 hostname: ${MAILCOW_HOSTNAME} networks: mailcow-network: ipv4_address: ${IPV4_NETWORK:-172.22.1}.250 aliases: - dovecot postfix-mailcow: image: mailcow/postfix:1.51 depends_on: - mysql-mailcow volumes: - ./data/hooks/postfix:/hooks - ./data/conf/postfix:/opt/postfix/conf - ./data/assets/ssl:/etc/ssl/mail/:ro - /srv/mailcow/postfix:/var/spool/postfix - /srv/mailcow/crypt:/var/lib/zeyple - /srv/mailcow/rspamd:/var/lib/rspamd - /srvlocal/mailcow/mysqld:/var/run/mysqld/ environment: - LOG_LINES=${LOG_LINES:-9999} - TZ=${TZ} - DBNAME=${DBNAME} - DBUSER=${DBUSER} - DBPASS=${DBPASS} - REDIS_SLAVEOF_IP=${REDIS_SLAVEOF_IP:-} - REDIS_SLAVEOF_PORT=${REDIS_SLAVEOF_PORT:-} cap_add: - NET_BIND_SERVICE ports: - "${SMTP_PORT:-25}:25" - "${SMTPS_PORT:-465}:465" - "${SUBMISSION_PORT:-587}:587" restart: unless-stopped dns: - ${IPV4_NETWORK:-172.22.1}.254 hostname: ${MAILCOW_HOSTNAME} networks: mailcow-network: aliases: - postfix memcached-mailcow: image: memcached:alpine restart: unless-stopped environment: - TZ=${TZ} networks: mailcow-network: aliases: - memcached nginx-mailcow: depends_on: - sogo-mailcow - php-fpm-mailcow - redis-mailcow image: nginx:mainline-alpine dns: - ${IPV4_NETWORK:-172.22.1}.254 command: /bin/sh -c "envsubst < /etc/nginx/conf.d/templates/listen_plain.template > /etc/nginx/conf.d/listen_plain.active && envsubst < /etc/nginx/conf.d/templates/listen_ssl.template > /etc/nginx/conf.d/listen_ssl.active && envsubst < /etc/nginx/conf.d/templates/server_name.template > /etc/nginx/conf.d/server_name.active && envsubst < /etc/nginx/conf.d/templates/sogo.template > /etc/nginx/conf.d/sogo.active && . /etc/nginx/conf.d/templates/sogo.auth_request.template.sh > /etc/nginx/conf.d/sogo_proxy_auth.active && . /etc/nginx/conf.d/templates/sites.template.sh > /etc/nginx/conf.d/sites.active && . /etc/nginx/conf.d/templates/sogo_eas.template.sh > /etc/nginx/conf.d/sogo_eas.active && nginx -qt && until ping phpfpm -c1 > /dev/null; do sleep 1; done && until ping sogo -c1 > /dev/null; do sleep 1; done && until ping redis -c1 > /dev/null; do sleep 1; done && until ping rspamd -c1 > /dev/null; do sleep 1; done && exec nginx -g 'daemon off;'" environment: - HTTPS_PORT=${HTTPS_PORT:-443} - HTTP_PORT=${HTTP_PORT:-80} - MAILCOW_HOSTNAME=${MAILCOW_HOSTNAME} - IPV4_NETWORK=${IPV4_NETWORK:-172.22.1} - TZ=${TZ} - SKIP_SOGO=${SKIP_SOGO:-n} - ALLOW_ADMIN_EMAIL_LOGIN=${ALLOW_ADMIN_EMAIL_LOGIN:-n} volumes: - ./data/web:/web:ro - ./data/conf/rspamd/dynmaps:/dynmaps:ro - ./data/assets/ssl/:/etc/ssl/mail/:ro - ./data/conf/nginx/:/etc/nginx/conf.d/:rw - ./data/conf/rspamd/meta_exporter:/meta_exporter:ro - /srv/mailcow/sogo_web:/usr/lib/GNUstep/SOGo/ ports: - "${HTTPS_BIND:-0.0.0.0}:${HTTPS_PORT:-443}:${HTTPS_PORT:-443}" - "${HTTP_BIND:-0.0.0.0}:${HTTP_PORT:-80}:${HTTP_PORT:-80}" restart: unless-stopped networks: mailcow-network: aliases: - nginx acme-mailcow: depends_on: - nginx-mailcow image: mailcow/acme:1.74 dns: - ${IPV4_NETWORK:-172.22.1}.254 environment: - LOG_LINES=${LOG_LINES:-9999} - ADDITIONAL_SAN=${ADDITIONAL_SAN} - MAILCOW_HOSTNAME=${MAILCOW_HOSTNAME} - DBNAME=${DBNAME} - DBUSER=${DBUSER} - DBPASS=${DBPASS} - SKIP_LETS_ENCRYPT=${SKIP_LETS_ENCRYPT:-n} - COMPOSE_PROJECT_NAME=${COMPOSE_PROJECT_NAME:-mailcow-dockerized} - DIRECTORY_URL=${DIRECTORY_URL:-} - ENABLE_SSL_SNI=${ENABLE_SSL_SNI:-n} - SKIP_IP_CHECK=${SKIP_IP_CHECK:-n} - SKIP_HTTP_VERIFICATION=${SKIP_HTTP_VERIFICATION:-n} - ONLY_MAILCOW_HOSTNAME=${ONLY_MAILCOW_HOSTNAME:-n} - LE_STAGING=${LE_STAGING:-n} - TZ=${TZ} - REDIS_SLAVEOF_IP=${REDIS_SLAVEOF_IP:-} - REDIS_SLAVEOF_PORT=${REDIS_SLAVEOF_PORT:-} - SNAT_TO_SOURCE=${SNAT_TO_SOURCE:-n} - SNAT6_TO_SOURCE=${SNAT6_TO_SOURCE:-n} volumes: - ./data/web/.well-known/acme-challenge:/var/www/acme:rw - ./data/assets/ssl:/var/lib/acme/:rw - ./data/assets/ssl-example:/var/lib/ssl-example/:ro - /srvlocal/mailcow/mysqld:/var/run/mysqld/ restart: unless-stopped networks: mailcow-network: aliases: - acme netfilter-mailcow: image: mailcow/netfilter:1.38 stop_grace_period: 30s depends_on: - dovecot-mailcow - postfix-mailcow - sogo-mailcow - php-fpm-mailcow - redis-mailcow restart: unless-stopped privileged: true environment: - TZ=${TZ} - IPV4_NETWORK=${IPV4_NETWORK:-172.22.1} - IPV6_NETWORK=${IPV6_NETWORK:-fd4d:6169:6c63:6f77::/64} - SNAT_TO_SOURCE=${SNAT_TO_SOURCE:-n} - SNAT6_TO_SOURCE=${SNAT6_TO_SOURCE:-n} - REDIS_SLAVEOF_IP=${REDIS_SLAVEOF_IP:-} - REDIS_SLAVEOF_PORT=${REDIS_SLAVEOF_PORT:-} network_mode: "host" volumes: - /lib/modules:/lib/modules:ro watchdog-mailcow: image: mailcow/watchdog:1.82 # Debug #command: /watchdog.sh dns: - ${IPV4_NETWORK:-172.22.1}.254 volumes: - /srv/mailcow/rspamd:/var/lib/rspamd - /srvlocal/mailcow/mysqld:/var/run/mysqld/ - /srv/mailcow/postfix:/var/spool/postfix - ./data/assets/ssl:/etc/ssl/mail/:ro restart: unless-stopped environment: - IPV6_NETWORK=${IPV6_NETWORK:-fd4d:6169:6c63:6f77::/64} - LOG_LINES=${LOG_LINES:-9999} - TZ=${TZ} - DBNAME=${DBNAME} - DBUSER=${DBUSER} - DBPASS=${DBPASS} - DBROOT=${DBROOT} - USE_WATCHDOG=${USE_WATCHDOG:-n} - WATCHDOG_NOTIFY_EMAIL=${WATCHDOG_NOTIFY_EMAIL} - WATCHDOG_NOTIFY_BAN=${WATCHDOG_NOTIFY_BAN:-y} - WATCHDOG_EXTERNAL_CHECKS=${WATCHDOG_EXTERNAL_CHECKS:-n} - WATCHDOG_MYSQL_REPLICATION_CHECKS=${WATCHDOG_MYSQL_REPLICATION_CHECKS:-n} - MAILCOW_HOSTNAME=${MAILCOW_HOSTNAME} - COMPOSE_PROJECT_NAME=${COMPOSE_PROJECT_NAME:-mailcow-dockerized} - IPV4_NETWORK=${IPV4_NETWORK:-172.22.1} - IP_BY_DOCKER_API=${IP_BY_DOCKER_API:-0} - CHECK_UNBOUND=${CHECK_UNBOUND:-1} - SKIP_CLAMD=${SKIP_CLAMD:-n} - SKIP_LETS_ENCRYPT=${SKIP_LETS_ENCRYPT:-n} - SKIP_SOGO=${SKIP_SOGO:-n} - HTTPS_PORT=${HTTPS_PORT:-443} - REDIS_SLAVEOF_IP=${REDIS_SLAVEOF_IP:-} - REDIS_SLAVEOF_PORT=${REDIS_SLAVEOF_PORT:-} - EXTERNAL_CHECKS_THRESHOLD=1 - NGINX_THRESHOLD=5 - UNBOUND_THRESHOLD=5 - REDIS_THRESHOLD=5 - MYSQL_THRESHOLD=5 - MYSQL_REPLICATION_THRESHOLD=1 - SOGO_THRESHOLD=3 - POSTFIX_THRESHOLD=8 - CLAMD_THRESHOLD=15 - DOVECOT_THRESHOLD=12 - DOVECOT_REPL_THRESHOLD=20 - PHPFPM_THRESHOLD=5 - RATELIMIT_THRESHOLD=1 - FAIL2BAN_THRESHOLD=1 - ACME_THRESHOLD=1 - IPV6NAT_THRESHOLD=1 - RSPAMD_THRESHOLD=5 - OLEFY_THRESHOLD=5 - MAILQ_THRESHOLD=20 - MAILQ_CRIT=30 networks: mailcow-network: aliases: - watchdog dockerapi-mailcow: image: mailcow/dockerapi:1.37 restart: unless-stopped oom_kill_disable: true dns: - ${IPV4_NETWORK:-172.22.1}.254 environment: - DBROOT=${DBROOT} - TZ=${TZ} volumes: - /var/run/docker.sock:/var/run/docker.sock:ro networks: mailcow-network: aliases: - dockerapi solr-mailcow: image: mailcow/solr:1.7 restart: unless-stopped volumes: - /srv/mailcow/solr/data:/opt/solr/server/solr/dovecot-fts/data ports: - "${SOLR_PORT:-127.0.0.1:18983}:8983" environment: - TZ=${TZ} - SOLR_HEAP=${SOLR_HEAP:-1024} - SKIP_SOLR=${SKIP_SOLR:-y} networks: mailcow-network: aliases: - solr olefy-mailcow: image: mailcow/olefy:1.3 restart: unless-stopped environment: - TZ=${TZ} - OLEFY_BINDADDRESS=0.0.0.0 - OLEFY_BINDPORT=10055 - OLEFY_TMPDIR=/tmp - OLEFY_PYTHON_PATH=/usr/bin/python3 - OLEFY_OLEVBA_PATH=/usr/bin/olevba3 - OLEFY_LOGLVL=20 - OLEFY_MINLENGTH=500 - OLEFY_DEL_TMP=1 networks: mailcow-network: aliases: - olefy ipv6nat-mailcow: depends_on: - unbound-mailcow - mysql-mailcow - redis-mailcow - clamd-mailcow - rspamd-mailcow - php-fpm-mailcow - sogo-mailcow - dovecot-mailcow - postfix-mailcow - memcached-mailcow - nginx-mailcow - acme-mailcow - netfilter-mailcow - watchdog-mailcow - dockerapi-mailcow - solr-mailcow environment: - TZ=${TZ} image: robbertkl/ipv6nat restart: unless-stopped privileged: true network_mode: "host" volumes: - /var/run/docker.sock:/var/run/docker.sock:ro - /lib/modules:/lib/modules:ro
networks: mailcow-network: driver: bridge driver_opts: com.docker.network.bridge.name: br-mailcow enable_ipv6: false ipam: driver: default config:
- subnet: ${IPV4_NETWORK:-172.22.1}.0/24
- subnet: ${IPV6_NETWORK:-fd4d:6169:6c63:6f77::/64}
/etc/fstab:
mail_data /mnt/mail_data 9p trans=virtio,version=9p2000.L,nobootwait,rw,_netdev 0 0 /mnt/mail_data/srv /srv auto bind 0 0 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.
There is an issue template. Please use it!
Hmm, wasn't sure if I should have used it - Not sure if my question is a bug - I was thinking it's more of a support question.... "Report a bug for this project. NOT to be used for support questions." - from the Bug Issue template
Your issue is probably very easy to fix: don't use ultra slow disks. I see this on servers with slow spinning disks, where the io-wait basically explodes when you start multiple containers.
I'm using NVME SSD.
Or do as your log suggests: Increase the timeout of docker-compose.
Already did, up to 5 minutes, as mentioned. Does this even look normal?
The issue tracker is not for support, correct. 👍 :)
No, it does not look normal then. I'm not sure if 9p mounts work. Never used them.
I tried again today, can confirm that it runs fine on local, but when volumes are mounted outside the VM image, it throws errors.
I did a docker-compose --verbose up -d
. This message tends to repeat itself:
compose.parallel.feed_queue: Pending: {<Service: acme-mailcow>}
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: {<Service: acme-mailcow>}
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: {<Service: acme-mailcow>}
compose.parallel.feed_queue: Pending: set()
Any hints on what I could look at to workaround this?
Secondary question - Not sure if it warrants its own topic: To keep VM images lean (around 32GB right now), I mount external drives for volumes. My setup couldn't use NFS as it complicates things with ZFS, docker and Hypervisors, so I'm using virtio-9p.
That's the context, and the question is: which volumes in the docker-compose.yml file are the ones that are most likely to take up a lot of space, as it gains a number of users, say 10-30, so I'll just focus on mounting those?
vmail-vol-1:
vmail-attachments-vol-1:
mysql-vol-1:
mysql-socket-vol-1:
redis-vol-1:
rspamd-vol-1:
solr-vol-1:
postfix-vol-1:
crypt-vol-1:
sogo-web-vol-1:
sogo-userdata-backup-vol-1:
Thanks!
vmail-vol-1 :)
Working just fine now with only vmail-vol-1: and vmail-attachments-vol-1: mounted on external drive.
My problem is solved, but still, this could be something to look into in the future for mailcow :)
I run into this same error several times when creating php-fpm and mysql containers.
On my qemu VM machine, I have bind-mounted the /srv/ directory onto a virtio 9p drive - refer to docker compose file. I've basically put every volume on this mounted device.
It works each time I do the docker-compose up -d with no pre-built files (i.e. I deleted it and it looks like a fresh install) However, I am testing it for stability and tried to do a docker-compose down, then docker-compose up again, but it always fails on the second up.
NOTE: I should note that when it works (i.e. the first time after a fresh install), I manage to get the whole system running - able to send and receive emails as expected. This issue is about testing the system for stability when I do a docker-compose down then up.
It always stops at php-fpm and mysql, with a timeout. I've tested setting timeout to 5mins, still no difference.
I suspect it could be due to the virtio drive - or maybe it isn't. And occassionally I see that mysql does not run when mysqld volume is on the mounted drive. So, I set the mysqld volume onto /srvlocal/ which is not on a mounted drive. Sometimes it works and sometimes it doesn't. Doing a docker-compose down then up doesn't always work as well as I'd hoped. There are too many variables here than I'm familiar with, and I'm still running into the same errors.
So my question is, which volumes are well-known to not work on a non-local drive?
docker-compose up: (sometimes, mysql works and postfix doesn't, but php-fpm always seems to be the culprit)
Also, once the above error shows, docker-compose down wouldn't work, and I cannot restart the docker service entirely. Even rebooting will timeout. I have to force stop the VM each time to recover.
mailcow.conf: [Summary: Only changed hostname, tz, watchdog email, compose name]
docker-compose.yml: [Summary: Changed volume locations only - removed named volumes to save space on qemu image. Mounted directly on virtio drive, which is the same physical drive the image sits on]
/etc/fstab: