Escaped characters in email address

[kingqueen3065]

This https://www.whatdotheyknow.com/request/foster_care_allowances_935#incoming-858469 suggests that Alaveteli escaped email address Firstname.O'Surname@midlothian.gov.uk to Firstname.O\'Surname@midlothian.gov.uk

though it may have been the receiving MTA that did this, also the email did eventually get through

[garethrees]

Looks like the followup was sent to the "bad" address https://www.whatdotheyknow.com/request/foster_care_allowances_935#outgoing-573269

[lizconlan]

Hmm, we are storing this properly and we generate a valid header in the form:

<From: Test User <fake@localhost>>, <To: O'Rly Books <O'Rly.Books@example.com>>

From examining the logs, the Exim MTA appears to have hedged its bets and sent to both forms of the address (e.g. O'Rly.Books and O\\'Rly.Books) which is a little weird but explains why we got two "Message accepted for delivery" log messages, and two autoresponses - an out of office and a bounceback.

Truncated, obfuscated initial mail log line:

from <[REDACTED]@whatdotheyknow.com> for O\\'Rly.Books@example.com O'Rly.Books@example.com

[lizconlan]

Not having much luck finding any references to this behaviour online, maybe it's acceptable? Could it be a stray rewriting rule?

[garethrees]

Might be something that @sagepe or someone in #sysadmin knows about – have you asked there?

[crowbot]

This is also occurring with an ampersand in the local part of an authority email. @sagepe has investigated and thinks that Alaveteli is submitting the escaped email address as an argument on the command line (exim extracts the unescaped one from the body of the email and sends to both). Looks like the culprit is a combination of the mail gem, which puts the unescaped address on the command line, and action_mailer, which is supplying the -t argument, causing exim to parse the command line for recipients.

Looks like the Rails action_mailer behaviour changes in release 4.2, so this may be resolved by https://github.com/mysociety/alaveteli/issues/2968.

[crowbot]

The change in 4.2 would also allow us to remove extract_addresses_remove_arguments=false from our exim config instructions.

[crowbot]

I think (see https://github.com/mikel/mail/issues/70) we might also be able to fix this by switching the delivery_method to exim from sendmail now, as the exim delivery method in mail does not put the destination address on the command line

[crowbot]

I think the reason the -t flag is there in the first place is in order to set the envelope-sender.

[mdeuk]

There's been a few instances recently where WDTK users have been unable to respond to messages from the London Borough of Hammersmith and Fulham as the email address LBHF is using contains an ampersand (as cited in #4642).

The email address in this instance is H&FInTouch @ domain, which seems to be correctly recognised by Alaveteli, however, when submitted to the MTA it is being processed as H\&FInTouch @ domain (e.g. a backslash is being introduced).

Two recent examples are: https://www.whatdotheyknow.com/request/junction_of_hammersmith_road_and?nocache=incoming-1257387#incoming-1257387 https://www.whatdotheyknow.com/request/tube_works_between_west_kensingt?nocache=incoming-1257416#incoming-1257416

[MattK1234]

A user encountered this issue last week on WhatDoTheyKnow, in request 586066 to Bournemouth, Christchurch and Poole Council.

This was picked up by one of our tracks that checks requests for bounce messages, so I have written to the user explaining how to work around it, by submitting the follow-up to the main contact address.

Another recent example is at request 501011.

[RichardTaylor]

At https://www.whatdotheyknow.com/request/current_future_plans_for_old_fir there is a response from an address starting:

Core.Ops&businfrastructureFOI

and following an attempt to reply to that, a bounce message referring to

Core.Ops\&businfrastructureFOI

[RichardTaylor]

Further example issue at

https://www.whatdotheyknow.com/request/documentation_relating_to_the_pr#incoming-1465283

The same authority and address is involved as at https://github.com/mysociety/alaveteli/issues/3465#issuecomment-537063719

[MattK1234]

Another instance of this issue in the request at https://www.whatdotheyknow.com/request/homelessness_main_questions_25

The reply email address began: hou&com.fois@

The mail logs refer to hou\&com.fois@

[mdeuk]

Another instance of this issue today at https://www.whatdotheyknow.com/request/list_of_hmos_and_the_address_of_5 (659750)

`Reporting-MTA: dns;LBHHLWECF0001.lbhf.gov.uk Received-From-MTA: dns;EUR03-VE1-obe.outbound.protection.outlook.com Arrival-Date: Mon, 20 Apr 2020 12:23:02 +0000

Final-Recipient: rfc822;H\&FInTouch@domain Action: failed Status: 5.1.10 Diagnostic-Code: smtp;550 5.1.10 RESOLVER.ADR.RecipientNotFound; Recipient not found by SMTP address lookup`

Unfortunately we also appear to get a side effect where the user's message shows as "delivered", but in actual fact, there has been a bounce back. I'd surmise this is because LBHF's email system (Microsoft 365 hybrid with EOP) is accepting the message without checking for validity of the address. I'll write that one up on its own later on.

[MattK1234]

I've created a PR to prevent replies being sent to the H&FInTouch email address as this one comes up semi-frequently

[MattK1234]

This issue was experienced on the request at https://www.whatdotheyknow.com/request/tvgs_processed_against_planning_65#outgoing-1058572

Intended email address: FOI-E&E@[domain]

Alaveteli sent to: FOI-E\&E@[domain]

[mdeuk]

This issue was experienced on the request at https://www.whatdotheyknow.com/request/tvgs_processed_against_planning_65#outgoing-1058572

Intended email address: FOI-E&E@[domain]

Alaveteli sent to: FOI-E\&E@[domain]

We've had a second report for this same email address, this time under request: https://www.whatdotheyknow.com/request/traffic_light_timings_for_pedest

I have created a PR which adds the FOI-E&E address to the workaround list.

[mdeuk]

We've had another instance, noted by @RichardTaylor in #5943 (wdtk request 700773)

On this occasion, the issue is with an email address of a Metropolitan Police Service official, however it otherwise follows the same pattern as the instances above (e.g. the email address appears to be properly formatted in the message headers, however when presented to the MTA there is some odd escaping going on).

[MattK1234]

Another instance of this issue at request 721462 on WDTK.

foi&dparequest@[DOMAIN]

PR raised at https://github.com/mysociety/whatdotheyknow-theme/pull/771 to prevent replies being sent to this email address. Will write to the user now to request they send their reply again.

[RichardTaylor]

Further example at:

https://www.whatdotheyknow.com/request/non_represented_nurses_and_midwi#incoming-1729573

Workaround proposed via

https://github.com/mysociety/whatdotheyknow-theme/issues/778

[RichardTaylor]

Further example of a public body using an & in an email address and this breaking functionality at:

https://www.whatdotheyknow.com/request/planned_works_3#incoming-1758519

[crowbot]

Logging another example https://www.whatdotheyknow.com/request/ministry_of_justice_it_contracto#outgoing-1190659

[RichardTaylor]

Another example of this issue:

https://www.whatdotheyknow.com/request/number_of_police_officers_at_eur#incoming-1885764

[sallytay]

Another example of this here: The occurrence of an apostrophe apparently breaking the ability to send a reply on the thread.

https://www.whatdotheyknow.com/request/staff_broken_down_by_declared_et#incoming-2005661

[RichardTaylor]

https://www.whatdotheyknow.com/request/contracted_printing_devices_and_1028#incoming-2062858

This is an example of this issue occurring in slightly different context, attempting to use an apostrophe in a request_address

It appears using an apostrophe in the request address resulted in Alaveteli unexpectedly sending mail to an address containing \' rather than just '.

[FOIMonkey]

Another example of this: https://www.whatdotheyknow.com/request/hague_judgments_convention#incoming-2127578

[HelenWDTK]

+1 We've had another example of this because the reply address contained an ampersand.

[HelenWDTK]

+1 Another one with an ampersand

[WilliamWDTK]

I'm not sure if this is completely related, but we have an issue with failed complete automatic redaction of an email address due to the presence of an apostrophe, where only the part after the apostrophe was automatically removed.

https://www.whatdotheyknow.com/request/pay_scales_and_job_titles_in_use#incoming-2260741

[HelenWDTK]

+1 Cropped up again today with an O' surname

[HelenWDTK]

+1 an & this time.

[gbp]

Attempted to replicate this issue using a gmail "+ address" with an &. I was able to receive mail into WDTK and send a followup from WDTK successfully.

[HelenWDTK]

Had another occurrence of this today with an &