Can't send emails & Most emails being temporarily rejected

[1n5aN1aC]

Classification

• [X] Question
• [X] Serious bug

Reproducibility

• [X] Always

Description

Ever since I updated my containers ~36 hours ago, (hardware/mailserver 81382b09484e) I have not been able to send any emails out. I updated again just now (hardware/mailserver d8e27be1671f) and same results.

As part of investigating that issue, I discovered that spamd is "temporarily rejecting" nearly all incoming emails. I believe the only emails I have gotten successfully since then are from YouTube & Steam.

Debugging information

rspamd webgui records all emails attempted to be sent and received, but provides no information about any of them except that they were "soft reject" My install was working fine and had not been updated, then broke when I updated all my dockers ~36 hours ago, so more recent commits could not cause this, while #275 #277 are both possible culprits.

docker logs mailserver   (when trying to send an email using rainloop)
2018-08-16T19:36:04.083440+00:00 mail postfix/submission/smtpd[736]: connect from unknown[172.21.0.1]
2018-08-16T19:36:04.194726+00:00 mail postfix/submission/smtpd[736]: Anonymous TLS connection established from unknown[172.21.0.1]: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
2018-08-16T19:36:04.236784+00:00 mail postfix/submission/smtpd[736]: 39BF1EC144E: client=unknown[172.21.0.1], sasl_method=PLAIN, sasl_username=myusernam@mydomain.net
2018-08-16T19:36:04.295056+00:00 mail postfix/authclean/cleanup[741]: 39BF1EC144E: replace: header Received: from webmail.mailmag.net (unknown [172.21.0.1])??(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))??(No client certificate requested)??by mail.mailmag.net (Postfix) with from unknown[172.21.0.1]; from=<myusernam@mydomain.net> to=<anotherusername@corban.edu> proto=ESMTP helo=<webmail.mailmag.net>: Received: from authenticated-user (mail.mailmag.net [127.0.0.1])??(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))??(No client certificate requested)??by mail.mailmag.net (Postfix) with ESMTPSA id 39BF1EC144E??for <jvillwock@corban.edu>; Thu, 16 Aug 2018 19:36:04 +0000 (UTC)
2018-08-16T19:36:04.295355+00:00 mail postfix/authclean/cleanup[741]: 39BF1EC144E: message-id=<8344ef49d0d4f099a4a3cb301fd0993c@mailmag.net>
2018-08-16T19:36:04.310071+00:00 mail postfix/authclean/cleanup[741]: 39BF1EC144E: milter-reject: END-OF-MESSAGE from unknown[172.21.0.1]: 4.7.1 Ratelimit \"user\" exceeded; from=<myusernam@mydomain.net> to=<anotherusername@corban.edu> proto=ESMTP helo=<webmail.mailmag.net>
2018-08-16T19:36:04.314309+00:00 mail postfix/submission/smtpd[736]: disconnect from unknown[172.21.0.1] ehlo=2 starttls=1 auth=1 mail=1 rcpt=1 data=0/1 commands=6/7

docker logs mailserver   (when receiving an email)
2018-08-16T19:38:46.639018+00:00 mail postfix/smtpd[766]: connect from mx-educause.informz.net[66.192.112.2]
2018-08-16T19:38:47.077166+00:00 mail postfix/smtpd[766]: Anonymous TLS connection established from mx-educause.informz.net[66.192.112.2]: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
2018-08-16T19:38:47.459298+00:00 mail postfix/smtpd[766]: 6FFFAEC144E: client=mx-educause.informz.net[66.192.112.2]
2018-08-16T19:38:47.706881+00:00 mail postfix/cleanup[769]: 6FFFAEC144E: message-id=<4742757755.4@informz.net>
2018-08-16T19:38:47.879518+00:00 mail postfix/cleanup[769]: 6FFFAEC144E: milter-reject: END-OF-MESSAGE from mx-educause.informz.net[66.192.112.2]: 4.7.1 Ratelimit \"to\" exceeded; from=<nde_4742757755.4@informz.net> to=<myusernam@mydomain.net> proto=ESMTP helo=<mx-educause.informz.net>

[hardware]

Probably related to this : https://github.com/rspamd/rspamd/issues/2360

A bug introduced in Rspamd 1.7.8 in ratelimit.lua has been fixed in Rspamd 1.7.9.

What version do you use ? This bug should not happen with the current builds on docker hub. 1.1-stable use rspamd 1.7.7 and 1.1-latest use rspamd 1.7.9.

Enable rspamd debug mode (only available in 1.1-latest for now) and search this lines :

(rspamd_proxy) <c7afe8>; lua; ratelimit.lua:394: ratelimit "to(1666:contact@domain.tld)" exceeded, (4 / 60): 0 (0:10.1221 dyn)
(rspamd_proxy) <c7afe8>; proxy; lua_task_set_pre_result: <20180721113120.374.33368@fcb246e58660>: set pre-result to soft reject: 'Ratelimit "to" exceeded'

mailserver:
  environment:
    DEBUG_MODE=rspamd

[1n5aN1aC]

Looks like it.

2018-08-16T21:35:13.113368+00:00 mail rspamd[751]: <d18980>; lua; ratelimit.lua:479: ratelimit "to(25000:myusername@mydomain.net)" exceeded, (4 / 0.016667): 0 (0:4.1535 dyn)
2018-08-16T21:35:13.113593+00:00 mail rspamd[751]: <d18980>; proxy; lua_task_set_pre_result: <1534453528-13556-1-git-send-email-bo.liu@linux.alibaba.com>: set pre-result to soft reject: 'Ratelimit "to" exceeded'

I ran rspamd --version inside the docker, and it says Rspamd daemon version 1.7.9 my docker image is currently on 516b57796ea2 hardware/mailserver:1.1-latest

Hmm, not sure where to go from here. I only have these directories mapped to the host / a volume:

    volumes:
      - ${VOLUMES_ROOT_PATH}/mail:/var/mail
      - ${VOLUMES_ROOT_PATH}/traefik/acme:/etc/letsencrypt/acme

[navossoc]

For an easy fix you can just disable the ratelimit until you figure what is happening:

    environment:
      - DISABLE_RATELIMITING=true              # Disable ratelimiting policy

[1n5aN1aC]

@navossoc Thanks, I'm getting emails now! So. Many.

@hardware Thanks, that was indeed the issue. Not sure how it could have happened though, as my comment above shows, I'm on rspamd 1.7.9, and I haven't done much customization on the container.

Here's my docker-compose.yml snippet.

  mailserver:
    image: hardware/mailserver:${MAILSERVER_DOCKER_TAG}
    container_name: mailserver
    restart: ${RESTART_MODE}
    domainname: ${DOMAIN}                      # Mail server A/MX/FQDN & reverse PTR = mail.domain.tld.
    hostname: ${HOSTNAME}
    labels:
      - traefik.enable=true
      - traefik.frontend.rule=Host:spam.${DOMAIN}
      - traefik.port=11334
      - traefik.docker.network=http_network
    ports:
      - "25:25"        # SMTP                - Required
      - "143:143"      # IMAP       STARTTLS - Optional - For webmails/desktop clients
    # - "465:465"      # SMTPS      SSL/TLS  - Optional - Enabled for compatibility reason, otherwise disabled
      - "587:587"      # Submission STARTTLS - Optional - For webmails/desktop clients
      - "993:993"      # IMAPS      SSL/TLS  - Optional - For webmails/desktop clients
      - "4190:4190"    # SIEVE      STARTTLS - Optional - Recommended for mail filtering
    environment:
      - DBPASS=${DATABASE_USER_PASSWORD}       # MariaDB database password (required)
      - RSPAMD_PASSWORD=${RSPAMD_PASSWORD}     # Rspamd WebUI password (required)
      - OPENDKIM_KEY_LENGTH=1024               # Apparently Namecheap only supports 255 character txt records
    # - ENABLE_POP3=true                       # Enable POP3 protocol
    # - ENABLE_FETCHMAIL=true                  # Enable fetchmail forwarding
    # - DISABLE_CLAMAV=true                    # Disable virus scanning
    # - DISABLE_SIGNING=true                   # Disable DKIM/ARC signing
    # - DISABLE_GREYLISTING=true               # Disable greylisting policy
      - DISABLE_RATELIMITING=true              # Disable ratelimiting policy
    #
    # Full list : https://github.com/hardware/mailserver#environment-variables
    volumes:
      - ${VOLUMES_ROOT_PATH}/mail:/var/mail
      - ${VOLUMES_ROOT_PATH}/traefik/acme:/etc/letsencrypt/acme
    depends_on:
      - mariadb
      - redis
    networks:
      - mail_network
      - http_network

[hardware]

@1n5aN1aC Can you try ratelimit.lua from the rspamd master branch ?

https://github.com/rspamd/rspamd/blob/master/src/plugins/lua/ratelimit.lua

mailserver:
  volumes:
    - ratelimit.lua:/usr/share/rspamd/lua/ratelimit.lua

Retry without DISABLE_RATELIMITING.

[1n5aN1aC]

That does appear to have resolved the issue. I then removed the ratelimit mapping etc, deleted the container, and put everything back up.

It appears as if the issue is resolved. Very strange as I haven't messed with any of the files inside the container, so I'm not sure how that could have happened.

Regardless, I'll be finding out whether or not it is resolved soon!

[hardware]

Very strange as I haven't messed with any of the files inside the container, so I'm not sure how that could have happened

This is probably not an issue on your side. Even if I have no more problems with version 1.7.9, that does not mean that the issue is definitely fixed for everybody. I think we have to wait for rspamd to stabilize his ratelimit algorithm.

I think I will stop updating it all the time, this is definitely not safe. But debian does not provide a recent and stable package, so I do not know how to do that.

[hardware]

I will disable the ratelimiting policy by default so that I can update the stable version and to avoid any future problem. Everyone will be free to activate it as needed with DISABLE_RATELIMITING=false and setting the thresholds manually.

[navossoc]

LGTM until rspamd makes it stable.