User cleaner does not detect all bounces correctly

[fosterfarrell9]

The UserCleaner worker which is run periodically to autodetect users who have mail adresses that are no longer valid does not detect many of the bounces.

[zebleck]

possibly solved through #414

[Splines]

@fosterfarrell9 Has this been fixed by #414?

[Splines]

Overview

The procedure for cleaning old users should be as follows:

• For any user that hasn't logged in in the last half year:
• Send an email à la "Your account will be deleted in 40 days if you don't log in again before then"
• 14 days before removal, send another warning email
• 2 days before approval, send an urgent warning email (high priority)
• If the user hasn't logged in before the deadline, delete them entirely from our system.

Intricacies

• Only delete generic users for right now. That is, not teachers or other users that might have created media in MaMpf. For those, we need to think about a different deletion strategy, e.g. assign the media to a generic "anonymous user" or "deleted user" such that the media can still exist even after the user deleted their account. We have to consider if there is a way that even generic users can create media in MaMpf?
• For generic users, what do we do with comments they've posted? Maybe write "deleted user" as author instead? Is it GDPR conform that the deleted user has no way to delete their posted comments in the future anymore?
• If we send the mail 40 days before the account is deleted, we should listen for bounces. If a mail bounces, send it again 1 day after the initial dispatch. We should use due dates in the body of the mail, e.g. "will be deleted at the ...." instead of "will be deleted in 40 days". This way, we don't have to write a new mail body when sending the bounced mail again. Do this procedure for the 40-days- and 14-days-before-deletion-mail. For the 1-day-before-deletion-mail, this won't have any sense as the user is already deleted by the time we resend the bounces mail. See below for how to detect bounced mails.

Helpful links for bounce detection via VERP

So far, the UserCleaner relies on certain body messages sent in bounced emails and parses them via regular expressions. This seems very fragile (there is no standard for such body messages, or at least not a standard that is widely adopted). Let's instead try to find an approach that is more robust.

• Your SMTP Envelope from address should not be the same as your RFC822 From Header address. Then any SMTP errors will go to the bounce address, but any real emails will go to the From address. Mailing List Managers use VERP to detect problems, by using a different envelope From address for each delivery. ~ StackOverflow

• Variable envelope return path (VERP) ~ Wikipedia We should probably use this with custom hashes such that no third-party person could just send an email to us with the recipient of another email in the header in order to delete that user even before the due date.

The only real disadvantage Wikipedia lists for this approach is the following_

Another problem with VERP (and with any automatic bounce handling scheme) is that there are MTAs on the Internet that fail to follow basic SMTP standards. VERP depends on the recipients' MTAs following the rule that bounces are sent to the envelope sender. This has been a standard requirement since the dawn of SMTP in 1982 (see RFC 821), but still there are MTAs that get it wrong, usually by bouncing to the address in the From: header.

I feel like we don't have to deal with yet another annoying case like this one. If the mail bounces AND the email server does not adhere to the SMPT standard, then I think it's not our fault and nobody can complain we didn't handle that case. We tried our best. It's goodwill from our side that we even consider resending the bounced mail again after 1 day, e.g. when the mail quota of the recipient is reached or when a mail server is down for some reason. We even send 3 mails in total before deleting the user enitrely.

• Somehow differentiate between hard/soft bounces? Is this even necessary for us to distinguish?
  • hard: address does not exist, feedback loop etc.
  • soft: mailbox quota etc