search

mailcow / mailcow-dockerized

mailcow: dockerized - 🐮 + 🐋 = 💕
https://mailcow.email
GNU General Public License v3.0
8.97k stars 1.18k forks source link

acme keeps incorrectly identifying a few random SSL certs as "orphans" and archiving them #6080

Closed andrewmarkforbes closed 1 month ago

andrewmarkforbes commented 2 months ago

Contribution guidelines

I've found a bug and checked that ...

Description

Acme keeps incorrectly identifying a handful of SSL certs on my server as "orphans", and archiving those SSL certs.

Every time I restart the acme container, most SSL certs on the server remain fine, but acme will archive a few different random SSL certs.

The server has a total of 125 domains. Not all of them need SSL certs, but most do. I have ADDITIONAL_SAN=mail.* and ENABLE_SSL_SNI=y

Occasionally acme will finish normally, and have no "orphans", but the next time it runs it will create a few more

I've only noticed this the last few days.

None of the domains are having DNS issues, they're all on cloudflare, and no DNS zones have been changed over the time of the problem.

Some of the domains were only registered inside the last month, so the SSL certs have never needed renewing yet

I've grepped the acme-mailcow log for a single domain to show the pattern, then pasted the last 50 general lines of acme-mailcow log to show how it usually finishes up.

Logs:

acme-mailcow-1  | Fri Sep 13 17:30:04 AEST 2024 - No A or AAAA record found for hostname mail.skyplay.com.au
acme-mailcow-1  | Fri Sep 13 17:30:04 AEST 2024 - Found A record for autodiscover.skyplay.com.au: 119.42.52.109
acme-mailcow-1  | Fri Sep 13 17:30:05 AEST 2024 - Found A record for autoconfig.skyplay.com.au: 119.42.52.109
acme-mailcow-1  | Fri Sep 13 17:31:46 AEST 2024 - Certificate /var/lib/acme/autodiscover.skyplay.com.au/cert.pem missing or changed domains 'autodiscover.skyplay.com.au autoconfig.skyplay.com.au' - start obtaining
acme-mailcow-1  | Fri Sep 13 17:31:47 AEST 2024 - Using command acme-tiny   --account-key /var/lib/acme/acme/account.pem --disable-check --csr /var/lib/acme/autodiscover.skyplay.com.au/acme.csr --acme-dir /var/www/acme/
acme-mailcow-1  | Found domains: autoconfig.skyplay.com.au, autodiscover.skyplay.com.au
acme-mailcow-1  | Already verified: autoconfig.skyplay.com.au, skipping...
acme-mailcow-1  | Already verified: autodiscover.skyplay.com.au, skipping...
acme-mailcow-1  | Fri Sep 13 17:31:58 AEST 2024 - Deploying certificate /var/lib/acme/autodiscover.skyplay.com.au/cert.pem...
acme-mailcow-1  | Fri Sep 13 17:32:08 AEST 2024 - Found orphaned certificate: mail.skyplay.com.au - archiving it at /var/lib/acme/backups/mail.skyplay.com.au/
acme-mailcow-1  | Fri Sep 13 17:48:02 AEST 2024 - Found A record for mail.skyplay.com.au: 119.42.52.109
acme-mailcow-1  | Fri Sep 13 17:48:02 AEST 2024 - Found A record for autodiscover.skyplay.com.au: 119.42.52.109
acme-mailcow-1  | Fri Sep 13 17:48:03 AEST 2024 - Found A record for autoconfig.skyplay.com.au: 119.42.52.109
acme-mailcow-1  | Fri Sep 13 17:49:27 AEST 2024 - Certificate /var/lib/acme/mail.skyplay.com.au/cert.pem missing or changed domains 'mail.skyplay.com.au autoconfig.skyplay.com.au autodiscover.skyplay.com.au' - start obtaining
acme-mailcow-1  | Fri Sep 13 17:49:27 AEST 2024 - Using command acme-tiny   --account-key /var/lib/acme/acme/account.pem --disable-check --csr /var/lib/acme/mail.skyplay.com.au/acme.csr --acme-dir /var/www/acme/
acme-mailcow-1  | Found domains: autoconfig.skyplay.com.au, autodiscover.skyplay.com.au, mail.skyplay.com.au
acme-mailcow-1  | Already verified: autoconfig.skyplay.com.au, skipping...
acme-mailcow-1  | Already verified: autodiscover.skyplay.com.au, skipping...
acme-mailcow-1  | Already verified: mail.skyplay.com.au, skipping...
acme-mailcow-1  | Fri Sep 13 17:49:39 AEST 2024 - Deploying certificate /var/lib/acme/mail.skyplay.com.au/cert.pem...
acme-mailcow-1  | Fri Sep 13 17:49:51 AEST 2024 - Found orphaned certificate: autodiscover.skyplay.com.au - archiving it at /var/lib/acme/backups/autodiscover.skyplay.com.au/
acme-mailcow-1  | Fri Sep 13 17:53:50 AEST 2024 - Found A record for mail.skyplay.com.au: 119.42.52.109
acme-mailcow-1  | Fri Sep 13 17:53:50 AEST 2024 - Found A record for autodiscover.skyplay.com.au: 119.42.52.109
acme-mailcow-1  | Fri Sep 13 17:53:50 AEST 2024 - Found A record for autoconfig.skyplay.com.au: 119.42.52.109
acme-mailcow-1  | Fri Sep 13 17:54:17 AEST 2024 - Certificate /var/lib/acme/mail.skyplay.com.au/cert.pem validation done, neither changed nor due for renewal.
acme-mailcow-1  | Fri Sep 13 18:04:16 AEST 2024 - Found A record for mail.skyplay.com.au: 119.42.52.109
acme-mailcow-1  | Fri Sep 13 18:04:17 AEST 2024 - Found A record for autodiscover.skyplay.com.au: 119.42.52.109
acme-mailcow-1  | Fri Sep 13 18:04:18 AEST 2024 - Found A record for autoconfig.skyplay.com.au: 119.42.52.109
acme-mailcow-1  | Fri Sep 13 18:04:37 AEST 2024 - Certificate /var/lib/acme/mail.skyplay.com.au/cert.pem validation done, neither changed nor due for renewal.
acme-mailcow-1  | Fri Sep 13 18:27:38 AEST 2024 - Found A record for mail.skyplay.com.au: 119.42.52.109
acme-mailcow-1  | Fri Sep 13 18:27:38 AEST 2024 - Found A record for autodiscover.skyplay.com.au: 119.42.52.109
acme-mailcow-1  | Fri Sep 13 18:27:38 AEST 2024 - Found A record for autoconfig.skyplay.com.au: 119.42.52.109
acme-mailcow-1  | Fri Sep 13 18:28:35 AEST 2024 - Certificate /var/lib/acme/mail.skyplay.com.au/cert.pem validation done, neither changed nor due for renewal.
acme-mailcow-1  | Fri Sep 13 19:02:10 AEST 2024 - Found A record for mail.skyplay.com.au: 119.42.52.109
acme-mailcow-1  | Fri Sep 13 19:02:11 AEST 2024 - Found A record for autodiscover.skyplay.com.au: 119.42.52.109
acme-mailcow-1  | Fri Sep 13 19:02:11 AEST 2024 - Found A record for autoconfig.skyplay.com.au: 119.42.52.109
acme-mailcow-1  | Fri Sep 13 19:02:39 AEST 2024 - Certificate /var/lib/acme/mail.skyplay.com.au/cert.pem validation done, neither changed nor due for renewal.

acme-mailcow-1  | Fri Sep 13 19:02:39 AEST 2024 - Certificate /var/lib/acme/mail.phb.org.au/cert.pem validation done, neither changed nor due for renewal.
acme-mailcow-1  | Fri Sep 13 19:02:39 AEST 2024 - Certificate /var/lib/acme/mail.pld.net.au/cert.pem validation done, neither changed nor due for renewal.
acme-mailcow-1  | Fri Sep 13 19:02:39 AEST 2024 - Certificate /var/lib/acme/mail.prizeplumbingsolutions.com.au/cert.pem validation done, neither changed nor due for renewal.
acme-mailcow-1  | Fri Sep 13 19:02:39 AEST 2024 - Certificate /var/lib/acme/mail.resilienceclinic.com.au/cert.pem validation done, neither changed nor due for renewal.
acme-mailcow-1  | Fri Sep 13 19:02:39 AEST 2024 - Certificate /var/lib/acme/mail.robbutlerpainting.com.au/cert.pem validation done, neither changed nor due for renewal.
acme-mailcow-1  | Fri Sep 13 19:02:39 AEST 2024 - Certificate /var/lib/acme/mail.rockcon.com.au/cert.pem validation done, neither changed nor due for renewal.
acme-mailcow-1  | Fri Sep 13 19:02:39 AEST 2024 - Certificate /var/lib/acme/mail.ruonline-mailer.com.au/cert.pem validation done, neither changed nor due for renewal.
acme-mailcow-1  | Fri Sep 13 19:02:39 AEST 2024 - Certificate /var/lib/acme/mail.skyplay.com.au/cert.pem validation done, neither changed nor due for renewal.
acme-mailcow-1  | Fri Sep 13 19:02:39 AEST 2024 - Certificate /var/lib/acme/mail.spcareconsulting.com.au/cert.pem validation done, neither changed nor due for renewal.
acme-mailcow-1  | Fri Sep 13 19:02:39 AEST 2024 - Certificate /var/lib/acme/mail.tatchell.com.au/cert.pem validation done, neither changed nor due for renewal.
acme-mailcow-1  | Fri Sep 13 19:02:39 AEST 2024 - Certificate /var/lib/acme/mail.templestowehealthhub.com.au/cert.pem validation done, neither changed nor due for renewal.
acme-mailcow-1  | Fri Sep 13 19:02:39 AEST 2024 - Certificate /var/lib/acme/mail.templestowepsychology.com.au/cert.pem validation done, neither changed nor due for renewal.
acme-mailcow-1  | Fri Sep 13 19:02:39 AEST 2024 - Certificate /var/lib/acme/mail.thehoppingmechanism.com.au/cert.pem validation done, neither changed nor due for renewal.
acme-mailcow-1  | Fri Sep 13 19:02:39 AEST 2024 - Certificate /var/lib/acme/mail.thomsonvalleysc.com.au/cert.pem validation done, neither changed nor due for renewal.
acme-mailcow-1  | Fri Sep 13 19:02:39 AEST 2024 - Certificate /var/lib/acme/mail.traditionaldelights.com.au/cert.pem validation done, neither changed nor due for renewal.
acme-mailcow-1  | Fri Sep 13 19:02:39 AEST 2024 - Certificate /var/lib/acme/mail.trekagroup.com.au/cert.pem validation done, neither changed nor due for renewal.
acme-mailcow-1  | Fri Sep 13 19:02:39 AEST 2024 - Certificate /var/lib/acme/mail.truebluebonefish.com.au/cert.pem validation done, neither changed nor due for renewal.
acme-mailcow-1  | Fri Sep 13 19:02:39 AEST 2024 - Certificate /var/lib/acme/mail.tvsc.au/cert.pem validation done, neither changed nor due for renewal.
acme-mailcow-1  | Fri Sep 13 19:02:39 AEST 2024 - Certificate /var/lib/acme/mail.weightmansmeats.com.au/cert.pem validation done, neither changed nor due for renewal.
acme-mailcow-1  | Fri Sep 13 19:02:39 AEST 2024 - Certificate /var/lib/acme/mail.wellbeingfocus.com.au/cert.pem validation done, neither changed nor due for renewal.
acme-mailcow-1  | Fri Sep 13 19:02:39 AEST 2024 - Certificate /var/lib/acme/mail.windrose.com.au/cert.pem validation done, neither changed nor due for renewal.
acme-mailcow-1  | Fri Sep 13 19:02:39 AEST 2024 - Found orphaned certificate: autodiscover.phb.org.au - archiving it at /var/lib/acme/backups/autodiscover.phb.org.au/
acme-mailcow-1  | Fri Sep 13 19:02:39 AEST 2024 - Found orphaned certificate: mail.jaylyndowns.com.au - archiving it at /var/lib/acme/backups/mail.jaylyndowns.com.au/
acme-mailcow-1  | Fri Sep 13 19:02:39 AEST 2024 - Reloading or restarting services... (1)
acme-mailcow-1  | Restarting 540fb709da54aff23510bbd994209fbfed3fc8885381d1acccaf1538b849e7fa...
acme-mailcow-1  | command completed successfully
acme-mailcow-1  | Restarting dd4162e498ebe2a684c5c7110c80492a6265cdb9c2b79131f775ecdc086357b0...
acme-mailcow-1  | command completed successfully
acme-mailcow-1  | Restarting 2d40460147eeca7b6adf9662787242a0dde9a749b245b9804362334a82cd852d...
acme-mailcow-1  | command completed successfully
acme-mailcow-1  | Fri Sep 13 19:02:56 AEST 2024 - Waiting for containers to settle...
acme-mailcow-1  | Fri Sep 13 19:03:06 AEST 2024 - Reloading or restarting services... (2)
acme-mailcow-1  | Restarting 540fb709da54aff23510bbd994209fbfed3fc8885381d1acccaf1538b849e7fa...
acme-mailcow-1  | command completed successfully
acme-mailcow-1  | Restarting dd4162e498ebe2a684c5c7110c80492a6265cdb9c2b79131f775ecdc086357b0...
acme-mailcow-1  | command completed successfully
acme-mailcow-1  | Restarting 2d40460147eeca7b6adf9662787242a0dde9a749b245b9804362334a82cd852d...
acme-mailcow-1  | command completed successfully
acme-mailcow-1  | Fri Sep 13 19:03:23 AEST 2024 - Waiting for containers to settle...
acme-mailcow-1  | Fri Sep 13 19:03:33 AEST 2024 - Reloading or restarting services... (3)
acme-mailcow-1  | Restarting 540fb709da54aff23510bbd994209fbfed3fc8885381d1acccaf1538b849e7fa...
acme-mailcow-1  | command completed successfully
acme-mailcow-1  | Restarting dd4162e498ebe2a684c5c7110c80492a6265cdb9c2b79131f775ecdc086357b0...
acme-mailcow-1  | command completed successfully
acme-mailcow-1  | Restarting 2d40460147eeca7b6adf9662787242a0dde9a749b245b9804362334a82cd852d...
acme-mailcow-1  | command completed successfully
acme-mailcow-1  | Fri Sep 13 19:03:49 AEST 2024 - Waiting for containers to settle...
acme-mailcow-1  | Fri Sep 13 19:04:00 AEST 2024 - Some services do return old end dates, something went wrong!
acme-mailcow-1  | OK
acme-mailcow-1  | Fri Sep 13 19:04:00 AEST 2024 - Certificates successfully requested and renewed where required, sleeping one day

Steps to reproduce:

1. add about 100 domains to mailcow

Which branch are you using?

master

Which architecture are you using?

x86

Operating System:

Debian Bookworm

Server/VM specifications:

11GB RAM, 4 CPU cores

Is Apparmor, SELinux or similar active?

no

Virtualization technology:

KVM

Docker version:

27.1.1

docker-compose version or docker compose version:

v2.29.1

mailcow version:

2024-08a

Reverse proxy:

N/A

Logs of git diff:

diff --git a/create_cold_standby.sh b/create_cold_standby.sh
index 924339af..25f57c16 100755
--- a/create_cold_standby.sh
+++ b/create_cold_standby.sh
@@ -2,6 +2,7 @@

 export REMOTE_SSH_KEY=/root/.ssh/id_rsa
 export REMOTE_SSH_PORT=22
-export REMOTE_SSH_HOST=my.remote.host
+#export REMOTE_SSH_HOST=netprescow.ruonline.com.au
+export REMOTE_SSH_HOST=npc

 /opt/mailcow-dockerized/helper-scripts/_cold-standby.sh
diff --git a/data/assets/ssl-example/cert.pem b/data/assets/ssl-example/cert.pem
index 96d16bec..78a9e0d0 100644
--- a/data/assets/ssl-example/cert.pem
+++ b/data/assets/ssl-example/cert.pem
@@ -1,19 +1,33 @@
 -----BEGIN CERTIFICATE-----
######## REDACTED #########
 -----END CERTIFICATE-----
diff --git a/data/assets/ssl-example/key.pem b/data/assets/ssl-example/key.pem
index cedf35a0..074569b4 100644
--- a/data/assets/ssl-example/key.pem
+++ b/data/assets/ssl-example/key.pem
@@ -1,27 +1,52 @@
------BEGIN RSA PRIVATE KEY-----
######## REDACTED #########
------END RSA PRIVATE KEY-----
+-----BEGIN PRIVATE KEY-----
######## REDACTED #########
+-----END PRIVATE KEY-----
diff --git a/data/conf/postfix/main.cf b/data/conf/postfix/main.cf
index 6a87f2ec..4e2d1741 100644
--- a/data/conf/postfix/main.cf
+++ b/data/conf/postfix/main.cf
@@ -173,3 +173,32 @@ parent_domain_matches_subdomains = debug_peer_list,fast_flush_domains,mynetworks

 # DO NOT EDIT ANYTHING BELOW #
 # Overrides #
+
+postscreen_dnsbl_sites = wl.mailspike.net=127.0.0.[18;19;20]*-2
+  hostkarma.junkemailfilter.com=127.0.0.1*-2
+  list.dnswl.org=127.0.[0..255].0*-2
+  list.dnswl.org=127.0.[0..255].1*-4
+  list.dnswl.org=127.0.[0..255].2*-6
+  list.dnswl.org=127.0.[0..255].3*-8
+  ix.dnsbl.manitu.net*2
+  bl.spamcop.net*2
+  bl.suomispam.net*2
+  hostkarma.junkemailfilter.com=127.0.0.2*3
+  hostkarma.junkemailfilter.com=127.0.0.4*2
+  hostkarma.junkemailfilter.com=127.0.1.2*1
+  backscatter.spameatingmonkey.net*2
+  bl.ipv6.spameatingmonkey.net*2
+  bl.spameatingmonkey.net*2
+  b.barracudacentral.org=127.0.0.2*7
+  bl.mailspike.net=127.0.0.2*5
+  bl.mailspike.net=127.0.0.[10;11;12]*4
+  zen.spamhaus.org=127.0.0.[10;11]*8
+  zen.spamhaus.org=127.0.0.[4..7]*6
+  zen.spamhaus.org=127.0.0.3*4
+  zen.spamhaus.org=127.0.0.2*3
+
+# User Overrides
+myhostname = mailcow.ruonline.com.au
+message_size_limit = 209715200
+#smtp_address_preference = ipv4
+#inet_protocols = ipv4
diff --git a/data/conf/rspamd/custom/global_mime_from_whitelist.map b/data/conf/rspamd/custom/global_mime_from_whitelist.map
index 3c872889..ef4cb079 100644
--- a/data/conf/rspamd/custom/global_mime_from_whitelist.map
+++ b/data/conf/rspamd/custom/global_mime_from_whitelist.map
@@ -1 +1,2 @@
-# /.+example\.com/i
+# /.+example\.com/i^M
+.*@ruonline\.com\.au
diff --git a/data/conf/rspamd/custom/global_smtp_from_whitelist.map b/data/conf/rspamd/custom/global_smtp_from_whitelist.map
index 3c872889..ef4cb079 100644
--- a/data/conf/rspamd/custom/global_smtp_from_whitelist.map
+++ b/data/conf/rspamd/custom/global_smtp_from_whitelist.map
@@ -1 +1,2 @@
-# /.+example\.com/i
+# /.+example\.com/i^M
+.*@ruonline\.com\.au
diff --git a/data/web/_status.502.html b/data/web/_status.502.html
index c8a9b702..25d52fa3 100644
--- a/data/web/_status.502.html
+++ b/data/web/_status.502.html
@@ -1,8 +1,20 @@
-<!DOCTYPE html>
+<!doctype html>
 <html>
   <head>
-    <title>Preparing</title>
+    <title>Mailcow Server Maintenance</title>
+    <meta charset="utf-8"/>
+    <meta name="robots" content="noindex"/>
     <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <style>
+      body { text-align: center; padding: 20px; font: 20px Helvetica, sans-serif; color: #efe8e8; }
+      @media (min-width: 768px){
+        body{ padding-top: 150px; }
+      }
+      h1 { font-size: 50px; }
+      article { display: block; text-align: left; max-width: 650px; margin: 0 auto; }
+      a { color: #dc8100; text-decoration: none; }
+      a:hover { color: #efe8e8; text-decoration: none; }
+    </style>
   </head>
   <body style='background-color:#fff;color:#333;width: 60%;margin: 0 auto;text-align:left;font-family: Verdana, "Lucida Sans Unicode", sans-serif'>
     <h1 style="color:#333;font-size:48px">What is happening?</h1>
@@ -23,3 +35,4 @@
     <br>Click to learn more about <a style="color:red;text-decoration:none;" href="https://docs.mailcow.email/#get-support" target="_blank">getting support.</a>
   </body>
 </html>
+
diff --git a/data/web/inc/vars.inc.php b/data/web/inc/vars.inc.php
index d3165b8a..40038846 100644
--- a/data/web/inc/vars.inc.php
+++ b/data/web/inc/vars.inc.php
@@ -183,7 +183,7 @@ $MAILBOX_DEFAULT_ATTRIBUTES['force_pw_update'] = false;
 $MAILBOX_DEFAULT_ATTRIBUTES['sogo_access'] = true;

 // Send notification when quarantine is not empty (never, hourly, daily, weekly)
-$MAILBOX_DEFAULT_ATTRIBUTES['quarantine_notification'] = 'hourly';
+$MAILBOX_DEFAULT_ATTRIBUTES['quarantine_notification'] = 'weekly';

 // Mailbox has IMAP access by default
 $MAILBOX_DEFAULT_ATTRIBUTES['imap_access'] = true;
@@ -201,7 +201,9 @@ $MAILBOX_DEFAULT_ATTRIBUTES['sieve_access'] = true;
 // "add_header" - mail that was put into the Junk folder
 // "reject" - mail that was rejected
 // "all" - mail that was rejected and put into the Junk folder
-$MAILBOX_DEFAULT_ATTRIBUTES['quarantine_category'] = 'reject';
+//$MAILBOX_DEFAULT_ATTRIBUTES['quarantine_category'] = 'reject';
+// edit 2023-05-06 - hopefully this isn't overwritten by a mailcow update
+$MAILBOX_DEFAULT_ATTRIBUTES['quarantine_category'] = 'all';

 // Default mailbox format, should not be changed unless you know exactly, what you do, keep the trailing ":"
 // Check dovecot.conf for further changes (e.g. shared namespace)
diff --git a/docker-compose.yml b/docker-compose.yml
index cf0a028f..ba23b82d 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -612,37 +612,6 @@ services:
         mailcow-network:
           aliases:
             - ofelia
-
-    ipv6nat-mailcow:
-      depends_on:
-        - unbound-mailcow
-        - mysql-mailcow
-        - redis-mailcow
-        - clamd-mailcow
-        - rspamd-mailcow
-        - php-fpm-mailcow
-        - sogo-mailcow
-        - dovecot-mailcow
-        - postfix-mailcow
-        - memcached-mailcow
-        - nginx-mailcow
-        - acme-mailcow
-        - netfilter-mailcow
-        - watchdog-mailcow
-        - dockerapi-mailcow
-        - solr-mailcow
-      environment:
-        - TZ=${TZ}
-      image: robbertkl/ipv6nat
-      security_opt:
-        - label=disable
-      restart: always
-      privileged: true
-      network_mode: "host"
-      volumes:
-        - /var/run/docker.sock:/var/run/docker.sock:ro
-        - /lib/modules:/lib/modules:ro
-
 networks:
   mailcow-network:
     driver: bridge

Logs of iptables -L -vn:

these iptables outputs made my bug report too long. Shall I include them as comments?

Logs of ip6tables -L -vn:

these iptables outputs made my bug report too long. Shall I include them as comments?

Logs of iptables -L -vn -t nat:

Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
3816K  257M DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
1318K  111M MASQUERADE  all  --  *      !br-mailcow  172.22.1.0/24        0.0.0.0/0           
    0     0 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0           
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.249         172.22.1.249         tcp dpt:6379
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.5           172.22.1.5           tcp dpt:8983
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.6           172.22.1.6           tcp dpt:3306
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.8           172.22.1.8           tcp dpt:80
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.8           172.22.1.8           tcp dpt:443
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:110
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:143
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:993
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:995
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:4190
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:12345
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.253         172.22.1.253         tcp dpt:25
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.253         172.22.1.253         tcp dpt:465
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.253         172.22.1.253         tcp dpt:587

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
 3634  218K RETURN     all  --  br-mailcow *       0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/0           
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:7654 to:172.22.1.249:6379
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:18983 to:172.22.1.5:8983
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:13306 to:172.22.1.6:3306
  107  5668 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80 to:172.22.1.8:80
  386 23172 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443 to:172.22.1.8:443
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:110 to:172.22.1.250:110
   55  3783 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:143 to:172.22.1.250:143
  874 55561 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:993 to:172.22.1.250:993
   10   544 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:995 to:172.22.1.250:995
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:4190 to:172.22.1.250:4190
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:19991 to:172.22.1.250:12345
  119  6908 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:25 to:172.22.1.253:25
   45  2706 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:465 to:172.22.1.253:465
   30  1768 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:587 to:172.22.1.253:587

Logs of ip6tables -L -vn -t nat:

Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 5434  427K DOCKER     all      *      *       ::/0                 ::/0                 ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER     all      *      *       ::/0                !::1                  ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 163K   16M MASQUERADE  all      *      !br-mailcow  fd4d:6169:6c63:6f77::/64  ::/0                
    0     0 MASQUERADE  all      *      !docker0  fd00:dead:beef:c0::/80  ::/0                
    0     0 MASQUERADE  all      *      docker0  ::/0                 ::/0                 ADDRTYPE match dst-type LOCAL
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::b  fd4d:6169:6c63:6f77::b  tcp dpt:80
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::b  fd4d:6169:6c63:6f77::b  tcp dpt:443
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::c  fd4d:6169:6c63:6f77::c  tcp dpt:110
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::c  fd4d:6169:6c63:6f77::c  tcp dpt:143
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::c  fd4d:6169:6c63:6f77::c  tcp dpt:993
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::c  fd4d:6169:6c63:6f77::c  tcp dpt:995
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::c  fd4d:6169:6c63:6f77::c  tcp dpt:4190
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::10  fd4d:6169:6c63:6f77::10  tcp dpt:25
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::10  fd4d:6169:6c63:6f77::10  tcp dpt:465
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::10  fd4d:6169:6c63:6f77::10  tcp dpt:587

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all      br-mailcow *       ::/0                 ::/0                
    0     0 RETURN     all      docker0 *       ::/0                 ::/0                
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:80 to:[fd4d:6169:6c63:6f77::b]:80
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:443 to:[fd4d:6169:6c63:6f77::b]:443
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:110 to:[fd4d:6169:6c63:6f77::c]:110
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:143 to:[fd4d:6169:6c63:6f77::c]:143
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:993 to:[fd4d:6169:6c63:6f77::c]:993
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:995 to:[fd4d:6169:6c63:6f77::c]:995
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:4190 to:[fd4d:6169:6c63:6f77::c]:4190
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:25 to:[fd4d:6169:6c63:6f77::10]:25
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:465 to:[fd4d:6169:6c63:6f77::10]:465
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:587 to:[fd4d:6169:6c63:6f77::10]:587

DNS check:

172.64.155.249
104.18.32.7
andrewmarkforbes commented 1 month ago

Does anyone have any ideas on this? It seemed to start with the latest mailcow update (version 2024-07ish to 2024-08a, and it's still happening. It seems to cycle through archiving ALL the additional SAN SSL certs on the server, a couple at a time, and once all the SSL certs have been archived and re-implemented, acme will run once or twice and report that "Certificates were successfully validated, no changes or renewals required, sleeping for another day." Then on the next run through, it'll start them all over again

andrewmarkforbes commented 1 month ago

After restarting the acme container a BUNCH more times, this issue seems to have resolved itself. The last 10 restarts have all resulted in "Certificates were successfully validated, no changes or renewals required, sleeping for another day."

Fingers crossed it stays that way.

I also noticed my server had IPv6 connectivity problems, which I just fixed, and acme is still now fine. Maybe that connectivity problem at the time of the last update triggered this whole thing. anyway, I guess this issue should be closed now.