Emails sent to stopped requests should follow RFC

[sebbacon]

Currently we just bounce emails to closed requests.

As per http://tools.ietf.org/html/rfc3834,

• Shouldn't bounce message back to Auto-Submitted
• Should check from address being replied to is valid
• Should set In-Reply-To and References fields
• Reconsider message content given that section in RFC

[olineham]

It is never possible to properly bounce email back to the true sender except at SMTP-time. Alavateli is contributing a significant amount of backscatter bounces to whichever poor sods email address spammers choose to use as a From address.

At the least, handle_rejected should be configurable and default to holding_pen or blackhole (at the moment it looks like it's hard coded to set bounce at the 6 month expiry). Ideally the "bounce" option should be removed entirely as (in my opinion) a misfeature.

Or (dreaming) the ultimate would be a postfix policy server which allows validation of request aliases at SMTP-time.

Please add new zealand label. Backscatter is affecting our email reputation as users at major providers are quite rightly marking our backscatter as spam itself.

[crowbot]

@olineham - handle_rejected is configurable from the admin page of a request, or is that not what you meant?

[olineham]

@crowbot That is a manual change, on each request. The default is not configurable as it's part of the database schema - https://github.com/mysociety/alaveteli/blob/3cacde6e382c4b8bc21e4503639a4f45e5e9f737/app/models/info_request.rb#L19

[crowbot]

@olineham ah, OK, I understand now.

[nigeljonez]

We are looking at this from a New Zealand perspective again as part of our effort to get the site updated, we'd like to get a change upstreamed rather than implement a local fix.

Our current thinking is that the only time 'bounce' is potentially RFC compliant (in a way that prevents backscatter) is if the incoming SMTP server has the ability to pull a list of addresses set to 'Allow new from nobody/reject with bounce', but most definitely 'bounce' should not be the default.

Our initial fix was going to be putting in a DB migration to change the default from bounce to holding_pen and add a DB view for Postfix (see nigeljonez/alaveteli@29f608a53143c68f14fb0639725fb686dae5a7d5). Seeing mysociety/alaveteli@fedd73a72c and associated rake tasks, I think for now simply setting the default action to holding_pen would be appropriate.

Would a pull request consisting of:

• a DB model change, default to holding_pen; and
• a setting via general.yml to the effect of DEFAULT_REJECTED_RESPONSE_ACTION, which if set, allows installations to keep bounce as a default action for new requests

be acceptable?

[lizconlan]

We're still looking into this but currently believe our usage is at least broadly in line with the spirit of the RFC but that we have confused the issue by using the terms "bounce" and "reject" to describe what the RFC calls an "automatic response". The intended outcome is to send a friendly response to messages sent to the closed request thread to inform the sender about what’s happened and provide details of how to contest that if they believe it should be reopened (e.g. an authority belatedly adding a response, or the request owner attempting to start a formal review process) rather than rejecting the email with an SMTP 554 response. (The time from last interaction to closure is configurable rather than hardcoded - added in version 0.23.0.0 so you should already have it - https://alaveteli.org/docs/running/requests/#old-requests-by-default-6-months-without-activity)

Then, as you've noted above, there's an advanced feature (first added in 0.25.0.0) to export a list of addresses for which incoming mail should be rejected at SMTP level - defined as closed requests that have met a (configurable) threshold of age of request and number of "rejected" messages, documented here: https://alaveteli.org/docs/running/handling_spam/#advanced-feature---rejection-of-incoming-messages-at-the-mta. In other words the point at which we're reasonably confident that there's unlikely to be any further genuine incoming email.

As such we're inclined to consider that these are going to be the correct defaults for most Alaveteli sites so we're likely to continue to at least recommend our current approach. (Although we are going to give some proper thought to making it more configurable as we can see that not everyone is going to have identical concerns to ours. In our case, pushing extra mail to the holding pen would increase the burden on the volunteer admins who look after the site, and silently blackholing late but legitimate correspondence would likely drive people away from the service.)

What we're going to do next is take a closer look at whether our automatic responses meet the RFC requirements or whether we need to make changes to reach full compliance.

We will get back to you with a fuller answer but sadly we can't give you a firm deadline as we're really stretched at the moment :(

[nigeljonez]

@lizconlan really, it's not a matter of the RFCs at the end of the day, it's the fact that Alaveteli cannot rely on the fact that the "From:" component is a verified sender, let it be due to lax SPF policies, or many other reasons.

Unless you are 100% confident that the sender of an email is the person named in "From:" the ONLY time you can send bounce messages is at SMTP-time - which delivers a reject/bounce as part of the SMTP chain of commands and puts the onus of poorly configured SMTP servers on those that run open relays.

Taking FYI as an example, until we made some extremely dirty changes to at least get part of it under control (i.e. getting away from the defaults in less than desirable ways because we didn't want to make code changes), what was happening was as follows...

1. Spamnet had acquired several of our requests addresses from PDF responses (direct text and/or pretty impressive OCR)
2. Forging "From:" headers to addresses hosted by QQ, hammered a closed request amongst others
3. Alaveteli received the emails with forged headers and opted to send the QQ email addresses with the preconfigured replies
4. Naturally the QQ users marked it as spam which results in RBL/Reputation issues

If Alaveteli forced decisions to happen at SMTP-time and accepted what came from it, then either, we'd be REJECTing the e-mail at SMTP time, and/or just accepting the response and putting it in holding_pen.

Now, this is just an extreme case, the point is there are various cases where the 'bounce' action within Alaveteli can be triggered, but it's our opinion that at the moment the currently situation needs to be looked at at fixed. Adherence to the RFC is not the only requirement, adherence to best practice is also required to be a good netizen when dealing with e-mail.

See https://en.wikipedia.org/wiki/Email_spoofing#The_effect_on_mailservers

[lizconlan]

@nigeljonez

Alaveteli cannot rely on the fact that the "From:" component is a verified sender

Yes, unfortunately by the time the message gets as far as the Rails application, it’s already been accepted by the SMTP gateway so it’s too late to reject the message (as you rightly point out - that needs to be done at SMTP-time) so the most the application code can do is decide whether to add the message to the request thread. What we’ve done to improve our confidence that it’s safe to reply is to configure our MTA with spam, malware and DNSBL checks to reduce the number of spoofed emails that make it through.

there are various cases where the 'bounce' action within Alaveteli can be triggered

We send 'bounce' messages in the following 2 cases:

The request has been automatically flagged as old (no activity for the number of months set in RESTRICT_NEW_RESPONSES_ON_OLD_REQUESTS_AFTER_MONTHS) which sets allow_new_responses_from to authority_only

The request has been automatically flagged as “very old” (no activity for RESTRICT_NEW_RESPONSES_ON_OLD_REQUESTS_AFTER_MONTHS x 4) which sets allow_new_responses_from to nobody

Plus the special case where the request has been manually updated to set allow_new_responses_from to be from 'nobody' or 'authority_only' (although in these cases, it seems reasonable to set handle_rejected to 'holding_pen' at the same time if that’s the desired effect).

As for a fix...

If you can get to parity with 0.25.0.0 - or otherwise pull in https://github.com/mysociety/alaveteli/commit/fedd73a72c - you’ll gain the ability to choose which requests you want the MTA to reject incoming mail for and, if you wish, can configure the REJECTED_THRESHOLD as 0 to start rejecting mail for old requests at SMTP time as soon as they’re flagged as closed.

And we’d be happy to accept a PR to allow the site to be configured to send messages that have been ‘rejected’ by the app to the holding pen for triage rather than sending auto responses, but we’d prefer to retain the existing default for new installs as we believe this is the correct behaviour for most reusers.

The approach we’d suggest is:

• add a new config key to general.yml (there’s an example of that here https://github.com/mysociety/alaveteli/pull/4498/commits/c43e3a0a91b2975ee3e74c6a3690c7643d393c39) to hold a configurable default value (make it 'bounce' in general.yml-example and lib/configuration.rb to retain our default then set your own general.yml to 'holding_pen')
• a database migration to allow InfoRequest.handle_rejected_responses to be null, and not have a default value
• add an after_initialize callback to InfoRequest to set null handle_rejected_responses to the value from the config key (probably worth checking that the value is allowable e.g. %w(bounce holding_pen blackhole).include?(value))
• write tests to prove:
  • the default value for a new InfoRequest is 'bounce'
  • the value for an existing InfoRequest that has been manually set to 'blackhole' doesn’t get overridden either with the standard default in place or when there is a config key set
  • the default value for a new InfoRequest when the config key is set to 'holding_pen' is 'holding_pen'

We’re continuing to look at this from our end to see if we’re currently missing anything that could help improve overall mail server reputation - for example, we’ve recently added a one-click unsubscribe link to the tracker emails to avoid people marking our emails as spam rather than going through a log-in-to-unsubscribe process https://github.com/mysociety/alaveteli/pull/4967 (this also allows us to unsubscribe people when we get their spam notifications through from Microsoft.) Not currently included in a release but seems stable - and useful - enough for us to be happy to recommend this as an approach.

Thanks for your patience with us! Sorry I left this so long without a reply :(

[garethrees]

Linking to https://github.com/mysociety/alaveteli/issues/4021.