question: DMS upgraded to a major release implicitly via Watchtower and is broken, what should I do?

mzedeler commented 3 weeks ago

📝 Preliminary Checks

[X] I tried searching for an existing issue and followed the debugging docs advice, but still need assistance.

👀 What Happened?

My server just pulled the latest version, which appears to be version 14, and now my log files are filling up with this:

mail          | 2024-06-09 17:43:09,864 WARN exited: fail2ban (exit status 255; not expected)

...and all mail services are down.

👟 Reproduction Steps

I am using v14 with the following settings:

OVERRIDE_HOSTNAME=<redacted>
ENABLE_CLAMAV=1
ENABLE_SPAMASSASSIN=1
ENABLE_MANAGESIEVE=1
POSTMASTER_ADDRESS=postmaster@<redacted>
ENABLE_POSTGREY=0
ENABLE_FAIL2BAN=1
SSL_TYPE=letsencrypt
TLS_LEVEL=intermediate
POSTSCREEN_ACTION=enforce

(I've just noticed that the above are not necessarily the same as the ones in docker-compose.yml below, but I don't think it matters.

🐋 DMS Version

v14.0.0

💻 Operating System and Architecture

Linux mail-server 5.4.0-65-generic #73-Ubuntu SMP Mon Jan 18 17:25:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

⚙️ Container configuration files

version: "3"
services:
  mail:
    image: mailserver/docker-mailserver:latest
    hostname: mail
    domainname: zedeler.dk
    container_name: mail
    cap_add:
      - NET_ADMIN
      - SYS_PTRACE
    ports:
      - "25:25"
      - "110:110"
      - "143:143"
      - "465:465"
      - "587:587"
      - "993:993"
      - "995:995"
      - "4190:4190"
    environment:
      - ENABLE_CLAMAV=1
      - ENABLE_SPAMASSASSIN=1
      - SPAMASSASSIN_SPAM_TO_INBOX=1
      - ENABLE_MANAGESIEVE=1
      - POSTMASTER_ADDRESS=postmaster@zedeler.dk
      - ENABLE_POSTGREY=1
      - ENABLE_FAIL2BAN=1
      - ONE_DIR=1
      - POSTFIX_MESSAGE_SIZE_LIMIT=26000000
      - SSL_TYPE=letsencrypt
    volumes:
      - /data/mail/data:/var/mail
      - /data/mail/state:/var/mail-state
      - /etc/letsencrypt:/etc/letsencrypt
      - /data/mail/config:/tmp/docker-mailserver
  watchtower:
    image: containrrr/watchtower
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command: --interval 30 mail

📜 Relevant log output

I tried to track this down by running fail2ban manually, and it seems to expect /var/run/fail2ban to exist, which it doesn't.

glime-ai[bot] commented 3 weeks ago

Please note that this response was created using generative AI, leveraging previously available information from the repository. This information may be helpful. However, it is our strong recommendation that you carefully consider the information before blindly applying it.

This response is generated only once - you will not be able to interact with the glime-ai.

It seems like the issue you're encountering with Fail2Ban is related to the missing /var/run/fail2ban directory. This directory is typically created when Fail2Ban starts, but it appears that it might not be happening in your case.

Here are a few steps you can take to troubleshoot and potentially resolve this issue:

Manually Create the Directory: You can try manually creating the directory inside the container and then restarting Fail2Ban. You can do this by executing the following commands:
```
docker exec -it mail bash
mkdir -p /var/run/fail2ban
supervisorctl restart fail2ban
```
Check Permissions: Ensure that the permissions for the /var/run directory allow Fail2Ban to create its directory. You can check and modify the permissions if necessary:
```
docker exec -it mail bash
chmod 755 /var/run
supervisorctl restart fail2ban
```
Review Configuration: Double-check your Fail2Ban configuration files to ensure there are no misconfigurations that might be causing Fail2Ban to fail on startup.
Logs and Debugging: Increase the log verbosity to get more detailed information about why Fail2Ban is failing. You can set the LOG_LEVEL environment variable to debug or trace:
```
environment:
 - LOG_LEVEL=debug
```
Update and Rebuild: Ensure that you are using the latest version of the Docker Mailserver image. Sometimes, issues are resolved in newer releases. You can pull the latest image and rebuild your container:
```
docker-compose pull
docker-compose up -d
```

If the issue persists, you might want to check the debugging documentation for more detailed troubleshooting steps.

This comment was generated by Glime.

georglauterbach commented 3 weeks ago

My server just pulled the latest version, which appears to be version 14, and now my log files are filling up with this:

Is that an unintended upgrade? We do not provide personal support in case you upgraded without carefully reading the changelog, as per our guidelines. I also saw "Watchtower" being used, and that's precisely why I'm asking.

Maintainer's time is the most precious resource this project has, hence my answer. To me, this just looks like Fail2Ban is not working on your system, and you did not bother to really investigate, instead opting to reach out for personal support - which we do not provide. I'm running :edge, and Fail2Ban is working quite nicely. I suggest running mkdir -p /var/run/fail2ban manually once.

polarathene commented 3 weeks ago

Never upgrade to a major release of DMS without checking our changelog.

We use semver tags for publishing releases, this is compatible with Watchtower when configured properly. I rewrote our docs regarding this, so be sure to review that and ensure you're using Watchtower correctly.

Config you should address (this isn't likely to fix your immediate issue)

ONE_DIR=1 is no longer a supported ENV in v14. The functionality is now implicit when a volume is mounted to /var/mail-state/.
version: "3" suggests you're on a version of Docker Compose that is many years old. Since Docker Compose v2+ you do not need to have a version field, it does absolutely nothing for you.
SYS_PTRACE capability is not required. We dropped this quite some time ago. Your compose.yaml is quite outdated.
hostname: mail + domainname: zedeler.dk is another example. Look at our current compose.yaml example in the repo (or view our docs), you should have all this set as hostname only. domainname is only for NIS domain config, it was a mistake by earlier maintainers.
image: mailserver/docker-mailserver:latest is a bad practice. You will encounter problems this way. Especially with Watchtower AFAIK. Use the proper semver tags. When major releases arrive, we sometimes break configuration compatibility, or expect manual changes on your end before you upgrade. We document these in the CHANGELOG.md.

DMS is not like other containers where you may have a smoother upgrade process. Major versions are usually at least once annually, care must be taken when upgrading.

No support can be given when the issue cannot be reproduced, you have not provided sufficient details for a fresh DMS deployment that can fail in the same way.

Linux mail-server 5.4.0-65

The 5.4 kernel was originally released in Nov 2019. Your system has not been upgraded in a while.

No wonder you're on an old Docker. You should address this, it's almost 5 years old. There have been improvements since that benefit Docker IIRC, especially with Docker itself and all the bug fixes you're missing.

andrewlow commented 3 weeks ago

FWIW - I ran into a similar problem.

Accessing the container and running the following inside the container

fail2ban-client -d

Was useful to tell me what was wrong.

Previously I had setup a config/fail2ban-fail2ban.cf file. Apparently the newer image contains a version of fail2ban which does not like the syntax I was using. It was complaining about missing a section header.

By removing this configuration file (and the copy that is placed into the container) - I was able to get fail2ban to start-up again just fine.

github-actions[bot] commented 1 day ago

This issue has become stale because it has been open for 20 days without activity. This issue will be closed in 10 days automatically unless:

a maintainer removes the meta/stale label or adds the stale-bot/ignore label
new activity occurs, such as a new comment

docker-mailserver / docker-mailserver