search

iredmail / dockerized

Official dockerized iRedMail.
https://www.iredmail.org/
267 stars 70 forks source link

Disk full, iredmail crashed, system not getting back up #88

Closed bluepuma77 closed 2 years ago

bluepuma77 commented 2 years ago

We had iredmail Docker running for about half a year, now the disk was full because of a sudden surge of emails. We made some space and restarted the iredmail docker container, but it's not working.

I see in /var/lib/mysql/mysqld.err that all users seem to have a wrong passwords and I see in /docker/entrypoints/settings.conf that the container is adding one or even two sets of new passwords on every restart. How can I get those passwords synced?

bluepuma77 commented 2 years ago

Just upgrading to the latest docker image does not work, different issues:

2021-12-11T08:52:16.726680181Z [iRedMail] Remove leftover pid files which may cause service fail to start.
2021-12-11T08:52:16.729678261Z [iRedMail] [Entrypoint] /docker/entrypoints/rsyslog.sh 
2021-12-11T08:52:16.744177049Z [iRedMail] [Entrypoint] /docker/entrypoints/cron.sh 
2021-12-11T08:52:16.746409835Z [iRedMail] [Entrypoint] /docker/entrypoints/mariadb.sh 
2021-12-11T08:52:16.751502631Z [iRedMail] Starting temporary MariaDB instance.
2021-12-11T08:52:16.768734314Z 2021-12-11  8:52:16 0 [Note] mysqld (mysqld 10.3.31-MariaDB-0ubuntu0.20.04.1-log) starting as process 213 ...
2021-12-11T08:52:16.772565716Z mysqld: Can't create file '/var/lib/mysql/mysqld.err' (errno: 13 "Permission denied")
2021-12-11T08:52:16.780511721Z 2021-12-11  8:52:16 0 [ERROR] mysqld: File '/var/lib/mysql/aria_log_control' not found (Errcode: 13 "Permission denied")
2021-12-11T08:52:16.780525126Z 2021-12-11  8:52:16 0 [ERROR] mysqld: Got error 'Can't open file' when trying to use aria control file '/var/lib/mysql/aria_log_control'
2021-12-11T08:52:16.780539262Z 2021-12-11  8:52:16 0 [ERROR] Plugin 'Aria' init function returned error.
2021-12-11T08:52:16.780544582Z 2021-12-11  8:52:16 0 [ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed.
2021-12-11T08:52:16.781152312Z 2021-12-11  8:52:16 0 [Note] InnoDB: Using Linux native AIO
2021-12-11T08:52:16.781163383Z 2021-12-11  8:52:16 0 [ERROR] InnoDB: The innodb_system data file 'ibdata1' must be writable
2021-12-11T08:52:16.781168813Z 2021-12-11  8:52:16 0 [ERROR] InnoDB: The innodb_system data file 'ibdata1' must be writable
2021-12-11T08:52:16.781174063Z 2021-12-11  8:52:16 0 [ERROR] Plugin 'InnoDB' init function returned error.
2021-12-11T08:52:16.781179172Z 2021-12-11  8:52:16 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2021-12-11T08:52:16.781259493Z 2021-12-11  8:52:16 0 [Note] Plugin 'FEEDBACK' is disabled.
2021-12-11T08:52:16.781331948Z 2021-12-11  8:52:16 0 [ERROR] mysqld: File '/var/lib/mysql/slow-query.log' not found (Errcode: 13 "Permission denied")
2021-12-11T08:52:16.781341056Z 2021-12-11  8:52:16 0 [ERROR] Could not use /var/lib/mysql/slow-query.log for logging (error 13). Turning logging off for the whole duration of the MariaDB server process. To turn it on again: fix the cause, shutdown the MariaDB server and restart it.
2021-12-11T08:52:16.781346757Z 2021-12-11  8:52:16 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2021-12-11T08:52:16.781351516Z 2021-12-11  8:52:16 0 [ERROR] Aborting
2021-12-11T08:52:16.781355984Z 
2021-12-11T08:52:16.783002180Z Warning: Memory not freed: 520
2021-12-11T08:52:46.943174137Z [iRedMail] Initialization failed. Please check /var/lib/mysql/mysqld.err for more details.
root@isy-email:/iredmail# docker logs iredmail2
[iRedMail] Remove leftover pid files which may cause service fail to start.
[iRedMail] [Entrypoint] /docker/entrypoints/rsyslog.sh 
[iRedMail] [Entrypoint] /docker/entrypoints/cron.sh 
[iRedMail] [Entrypoint] /docker/entrypoints/mariadb.sh 
[iRedMail] Starting temporary MariaDB instance.
2021-12-11  8:52:16 0 [Note] mysqld (mysqld 10.3.31-MariaDB-0ubuntu0.20.04.1-log) starting as process 213 ...
mysqld: Can't create file '/var/lib/mysql/mysqld.err' (errno: 13 "Permission denied")
2021-12-11  8:52:16 0 [ERROR] mysqld: File '/var/lib/mysql/aria_log_control' not found (Errcode: 13 "Permission denied")
2021-12-11  8:52:16 0 [ERROR] mysqld: Got error 'Can't open file' when trying to use aria control file '/var/lib/mysql/aria_log_control'
2021-12-11  8:52:16 0 [ERROR] Plugin 'Aria' init function returned error.
2021-12-11  8:52:16 0 [ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed.
2021-12-11  8:52:16 0 [Note] InnoDB: Using Linux native AIO
2021-12-11  8:52:16 0 [ERROR] InnoDB: The innodb_system data file 'ibdata1' must be writable
2021-12-11  8:52:16 0 [ERROR] InnoDB: The innodb_system data file 'ibdata1' must be writable
2021-12-11  8:52:16 0 [ERROR] Plugin 'InnoDB' init function returned error.
2021-12-11  8:52:16 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2021-12-11  8:52:16 0 [Note] Plugin 'FEEDBACK' is disabled.
2021-12-11  8:52:16 0 [ERROR] mysqld: File '/var/lib/mysql/slow-query.log' not found (Errcode: 13 "Permission denied")
2021-12-11  8:52:16 0 [ERROR] Could not use /var/lib/mysql/slow-query.log for logging (error 13). Turning logging off for the whole duration of the MariaDB server process. To turn it on again: fix the cause, shutdown the MariaDB server and restart it.
2021-12-11  8:52:16 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2021-12-11  8:52:16 0 [ERROR] Aborting

Warning: Memory not freed: 520
[iRedMail] Initialization failed. Please check /var/lib/mysql/mysqld.err for more details.
bluepuma77 commented 2 years ago

When I need to restart with a completely new iredmail container setup, then I need to deal with the legacy:

  1. How to re-use old SPF, DKIM, DMARC records within iredmail instead of creating new ones?
  2. How to restore email users with passwords?
  3. How to restore emails?
bluepuma77 commented 2 years ago

fail2ban is working fine, it bans my email sending server now ;-) 

# fail2ban-client status postfix
Status for the jail: postfix
|- Filter
|  |- Currently failed: 1
|  |- Total failed: 18
|  `- File list:    /var/log/mail.log
`- Actions
   |- Currently banned: 1
   |- Total banned: 1
   `- Banned IP list:   1.2.3.4

Stop it via fail2ban-client stop

bluepuma77 commented 2 years ago

Deleting all password in /docker/entrypoints/settings.conf results in a single new set of passwords.

That works to login with postmaster into /iredadmin website. It shows my domain, but edit page is not showing?!

Email users can still not login into roundcube and can not send emails via SMTP.

In the /var/lib/mysql/mysqld.err logs I see:

[Warning] Access denied for user 'vmail'@'localhost' (using password: YES)
[Warning] Access denied for user 'sa_bayes'@'localhost' (using password: YES)
[Warning] Access denied for user 'amavisd'@'localhost' (using password: YES)
[Warning] Aborted connection 39 to db: 'vmail' user: 'vmailadmin' host: 'localhost' (Got an error reading communication packets)
bluepuma77 commented 2 years ago

VMAIL_DB_PASSWORD set in /docker/entrypoints/settings.conf is the same as in .my.cnf-vmail.

But the password is different in all configuration files in /etc/postfix/mysql/* !

Quick fix to get at least SMTP up and running again:

cd /etc/postfix/mysql/
mkdir old
cp *.cf old/
sed -i 's/OLD-HASH-IN-FILES/NEW-HASH-FROM-CONF/g' *.cf
postfix reload

Just don't restart your container ;-)

bluepuma77 commented 2 years ago

The quick fix won't survive a system reboot, so at some point I need to migrate to the latest version.

How can I do that? Do I see "Permission denied" errors because of user (right) changes?

Or should I just dump the old database and re-import it into the latest image container?

Is there a way to keep the old SPF, DKIM and DMARC records?

bluepuma77 commented 2 years ago

The database file permission issue was mentioned before: https://github.com/iredmail/dockerized/issues/67.

Strangely enough I thought I set all mysql files to a+rw on the host, so I didn't expect any issues.

bluepuma77 commented 2 years ago

iRedAdmin has the latest passwords in /opt/www/iredadmin/settings.py.

Dovecot has the latest password in /etc/dovecot/dovecot-mysql.conf.

Roundcube has the latest password in /opt/www/roundcubemail/config/config.inc.php.

SpamAssassin has an old password in /etc/mail/spamassassin/local.cf. Update.

Amavisd has an old password in /etc/amavisd.conf. Update and reload with /usr/sbin/amavisd reload.

The server is under continuous load and I see MySQL/MariaDB error messages:

[Warning] Aborted connection 111511 to db: 'vmail' user: 'vmailadmin' host: 'localhost' (Got an error reading communication packets)

Current situation:

  1. SMTP sending works
  2. iRedAdmin can login, but not load domain details
  3. Roundcube can not login users
bluepuma77 commented 2 years ago

Too bad docker hub does not have the old images with version numbers available. Maybe I could have gotten a version that does not regenerate all keys on every startup and does not have the database changes.

Preparing for a fresh install: DKIM is referenced in /etc/amavisd.conf and stored in /opt/iredmail/custom/amavisd/dkim/DOMAIN.TLD.pem.

bluepuma77 commented 2 years ago

My litte incident took me a full day, I finally I decided to setup a new dockerized iRedMail in a different host directory.

I just copied the old iredmail-docker.conf and /opt/iredmail/custom/amavisd/dkim/DOMAIN.TLD.pem.

New iRedMail then did a full setup, I could re-use old postmaster password and re-created my 10 email users. We use iredmail for transactional emails.

I knew I still had some emails in queue and old emails in the mailbox, so I needed access to the old container. Interestingly enough you can use docker commit iredmail iredmail-old-img to create an image from the old container (which is not available from docker hub anymore). Then you can start the old environment with reduced and remapped ports docker run -name iredmail-old -p 8080:80 -p 8443:443 ... iredmail-old-img.

I removed all passwords from bottom of /docker/entrypoints/settings.conf, restarted the container again. Updated the password in postfix (cd /etc/postfix/mysql/; sed -i 's/OLD-HASH-IN-FILES/NEW-HASH-FROM-CONF/g' *.cf; postfix reload) and amavis (vi /etc/amavisd.conf; /usr/sbin/amavisd reload). Restarted the queue (postfix flush) and watched all the old queued mails being sent (tail -f /var/log/mail.log). Via https://domain.tld:8443 we still have access to the old mailbox and can clean it up.

New iRedMail seems not to overwrite all passwords on every restart (or it changes them in every required file), so we should be fine as long as I watch the available disk space ;-)

bluepuma77 commented 2 years ago

Mail-Tester is giving me a DKIM warning, need to investigate. Seems the latest iRedMail image changed things around. /etc/amavisd.conf is not there anymore, instead it's /etc/amavis/conf.d/50-user. To show the active DKIM key, use amavisd-new -c /etc/amavis/conf.d/50-user showkeys within container.

Turns out the DOMAIN.TLD.pem file was modified during setup, although I had provided the old existing one. Now I had to update the DNS entry. Seems we have a good enough reputation that no emails got bounced in the meantime.

PS: Very helpful iRedMail docs: File locations and Setup DNS.

bluepuma77 commented 2 years ago

Emails still piling up. Speed of email sending is too slow.

Two options:

  1. Process more emails concurrently
  2. Disable spam virus scanning for outgoing mails

Use /etc/init.d/amavis restart and postfix stop (seems postfix start is done automatically).

I wonder why iRedMail has only 1 postfix and 1 amavis process by default.