PhysioNet e-mail sent to spam folder

[jraffa]

Received a non-critical e-mail about a physionet project that ended up in my spam folder. Looks like it might be a known issue, but thought I'd fwd headers. Only noticed because I was waiting for a different e-mail today, and gave that folder a look.

Maybe DKIM would help?

Received: from oc11expo22.exchange.mit.edu (18.9.4.84) by
 oc11expo22.exchange.mit.edu (18.9.4.84) with Microsoft SMTP Server (TLS) id
 15.0.1365.1 via Mailbox Transport; Tue, 30 Jun 2020 16:26:42 -0400
Received: from oc11exhyb6.exchange.mit.edu (18.9.1.111) by
 oc11expo22.exchange.mit.edu (18.9.4.84) with Microsoft SMTP Server (TLS) id
 15.0.1365.1; Tue, 30 Jun 2020 16:26:41 -0400
Received: from NAM02-CY1-obe.outbound.protection.outlook.com (104.47.37.53) by
 oc11exhyb6.exchange.mit.edu (18.9.1.111) with Microsoft SMTP Server (TLS) id
 15.0.1395.4 via Frontend Transport; Tue, 30 Jun 2020 16:26:41 -0400
Received: from CO2PR04CA0055.namprd04.prod.outlook.com (2603:10b6:102:1::23)
 by DM6PR01MB4476.prod.exchangelabs.com (2603:10b6:5:7a::20) with Microsoft
 SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
 15.20.3131.25; Tue, 30 Jun 2020 20:26:40 +0000
Received: from CO1NAM03FT018.eop-NAM03.prod.protection.outlook.com
 (2603:10b6:102:1:cafe::cc) by CO2PR04CA0055.outlook.office365.com
 (2603:10b6:102:1::23) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3131.21 via Frontend
 Transport; Tue, 30 Jun 2020 20:26:40 +0000
Authentication-Results: spf=pass (sender IP is 128.30.29.50)
 smtp.mailfrom=physionet.org; mit.edu; dkim=none (message not signed)
 header.d=none;mit.edu; dmarc=bestguesspass action=none
 header.from=physionet.org;compauth=pass reason=109
Received-SPF: Pass (protection.outlook.com: domain of physionet.org designates
 128.30.29.50 as permitted sender) receiver=protection.outlook.com;
 client-ip=128.30.29.50; helo=mail-csail.ecg.mit.edu;
Received: from mail-csail.ecg.mit.edu (128.30.29.50) by
 CO1NAM03FT018.mail.protection.outlook.com (10.152.80.174) with Microsoft SMTP
 Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
 15.20.3131.20 via Frontend Transport; Tue, 30 Jun 2020 20:26:39 +0000
Received: from physionet-production.ecg.mit.edu (physionet-production.ecg.mit.edu [192.168.11.103])
    by mail-csail.ecg.mit.edu (Postfix) with ESMTP id DA9BC9FDCA
    for <snip@mit.edu>; Tue, 30 Jun 2020 16:26:38 -0400 (EDT)
Received: from physionet-production.ecg.mit.edu (localhost [127.0.0.1])
    by physionet-production.ecg.mit.edu (Postfix) with ESMTP id CBF6D33C
    for <snip@mit.edu>; Tue, 30 Jun 2020 16:26:38 -0400 (EDT)
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Subject: Authorship invitation accepted for project: The Global Open Source
 Severity of
 Illness Score (GOSSIS) Dataset
From: PhysioNet Automated System <noreply@physionet.org>
To: <snip@mit.edu>
Date: Tue, 30 Jun 2020 20:26:38 +0000
Message-ID: <159354879883.29685.11078396368536037797@physionet-production.ecg.mit.edu>
Return-Path: noreply@physionet.org
X-EOPAttributedMessage: 0
X-EOPTenantAttributedMessage: 64afd9ba-0ecf-4acf-bc36-935f6235ba8b:0
X-Forefront-Antispam-Report: CIP:128.30.29.50;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail-csail.ecg.mit.edu;PTR:heimdallr.csail.mit.edu;CAT:NONE;SFTY:;SFS:(4636009)(26005)(7126003)(336012)(356005)(7696005)(86362001)(4744005)(19627235002)(5660300002)(4006050)(75640400001)(8676002)(1096003)(6266002)(55016002)(9686003)(83380400001)(6916009);DIR:INB;SFP:;
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 6fbbe891-b74c-48d5-dd83-08d81d33e541
X-MS-TrafficTypeDiagnostic: DM6PR01MB4476:
X-LD-Processed: 64afd9ba-0ecf-4acf-bc36-935f6235ba8b,ExtAddr
X-MS-Exchange-AtpMessageProperties: SA
X-MS-Oob-TLC-OOBClassifiers: OLM:3968;
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: =?utf-8?B?MitKeG5CVldiaUxvQlE1SnF0aDBJTDVNc1FWTkpDM002RFI4eno5VTlleHc5?=
 =?utf-8?B?TmtobnBpK2F6U2dyeC83NTE1dVlvNUJ3UjZhZmtKQk5KWVNIQkJNcTc5RHFr?=
 =?utf-8?B?WG5UYzI4R3QzZnMzUnM0K3pDSmZPNTRRcThvSmxwU25idWJqdnpFRFBBSGNJ?=
 =?utf-8?B?eHVtQjRlMGR2eWlKem05ZUZxYk9ZelE2VHdETUxuVVN1U3JJNDRWanlQZm1O?=
 =?utf-8?B?cjRsUjViQmIzcFNmcjN2Ti95cGozZkhEcWR1ZE0zSmFJWFhJekwxb0JhUUQz?=
 =?utf-8?B?WU85eWtFMkNzeEtVVVJ6WVFPL0laTWhPNzVlaVRJckwxa1ZNcjk1b2MvTC90?=
 =?utf-8?B?cXd6WDl1K3ZMOWxPelRUSlpMZDNxLy9pS0ZjU3o1SVBaditSK3dkVExZRzNa?=
 =?utf-8?B?Q2sydTNYeVJ3QURaV0txYUt1QnUrUDN0bWNEcW5Xdm9WaWdRTUpKTEw0bldH?=
 =?utf-8?B?ek40cVJ1VnYxMXpIOWxqZndFSWlxUnd1R1VUOXBJMGlZWVI1Z2MzZFZ2S2ho?=
 =?utf-8?B?Nmd4L2ZrallZZkZVWWJPcHhWdWlDN2cyMFpFOUxOUkx5b2JCL2FhQXV0Zksy?=
 =?utf-8?B?bSt6SGMvZkNURTMxbEJPMHhVb0dEQzV4R3E5bVB4K05Sb3lyajYwRDNEKytk?=
 =?utf-8?B?b3FURXcxT1EvazNZRWszSXNLUWljWUlNUElVZkwzSHEvT0dySDZRUFNaZkMv?=
 =?utf-8?B?Zm9uU0F5ckRza2xVeENhdjZyTFlBUlFka1ZSSDBwcEZTclhoTW1QOEtEVFp1?=
 =?utf-8?B?T0xuR1dBRlhKakZIOThTN2hIOTZMS21WUlFrZzNwdUNZcHIxZGhXZThFbmhJ?=
 =?utf-8?B?Y0JrUllQdTNUM3A1ai9MS3dtZTRpbm5KbU9ndlZGNzc4aW1lb2FJT0JWSTlR?=
 =?utf-8?B?Q1paZFAxT05YbTdtcUEvOSsxaEVvenBNNXo2cGowWVk4bDNZUGJ1L0pOUmRW?=
 =?utf-8?B?NVA2amZ2STdLYmxOSXlyOStjNzFUd2Ryb2d1MTBjeGNDeER6QzRubytTY0pQ?=
 =?utf-8?B?QUNuMGd4ZW02ejF6dmZUN2ZEVDhpZTRwSUc2SmtKYjA0SDNDUWwvOWU5dDdx?=
 =?utf-8?B?NlFJSTI4RldIVDlQU292VkRUbXF1cUFuaWZjZWduSXo2VnZ6S3pmK0dUWFNW?=
 =?utf-8?B?aGJJWkJLZVNIWEtwYWtQcXplSTR5TjN5TkVyTUV1bm82TWp0RnZkVmpoVEgz?=
 =?utf-8?B?RE5GcHllR0kwMVg1Sm0ybFR4S05GNnoyWk9vK2FZMkh5bUxxSm16WW1XSHg0?=
 =?utf-8?B?c2dOZDJNUVNCWHM1NzRSQm5XVWpuVVF1UjAwL0tONHlNT0FmVGhEZUN5TXJB?=
 =?utf-8?B?RUo0RU5BNmxyaE9XbDU2aTZQV1VNc3AxSVd1ZEc1NkpQYWtyS3N4T0NnM0da?=
 =?utf-8?B?eXoyOTRvcDRBWTlNb3kwYXhQbTBuaFVaRTFRMGd1aWV5MXRHbGNrUzFCVTk4?=
 =?utf-8?B?TXoyK1liTzFZMFhVU25JU0hjMWo1Mks5bE1wYTlDM0I4aUp0aWZUVG8yMzdE?=
 =?utf-8?B?WmQwZjNhUFpENWZEYUJMNHdFb3dqL0RKU0J0WkRIc3RLeFJXV2k2L05Gemtq?=
 =?utf-8?Q?m6Zy6p5fq0xv/aS5gV/d9fF6nHp8UZ4Zbn4I/cRU5f1pqF?=
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Jun 2020 20:26:39.7775
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 6fbbe891-b74c-48d5-dd83-08d81d33e541
X-MS-Exchange-CrossTenant-Id: 64afd9ba-0ecf-4acf-bc36-935f6235ba8b
X-MS-Exchange-CrossTenant-AuthSource: CO1NAM03FT018.eop-NAM03.prod.protection.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: Internet
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR01MB4476
X-OrganizationHeadersPreserved: DM6PR01MB4476.prod.exchangelabs.com
X-MS-Exchange-Organization-SCL: 1
X-CrossPremisesHeadersPromoted: oc11exhyb6.exchange.mit.edu
X-CrossPremisesHeadersFiltered: oc11exhyb6.exchange.mit.edu
X-MS-Exchange-Organization-Network-Message-Id: 3260d375-8c53-47c8-1964-08d81d33e66e
X-MS-Exchange-Organization-AuthSource:
    CO1NAM03FT018.eop-NAM03.prod.protection.outlook.com
X-MS-Exchange-Organization-AuthAs: Anonymous
X-OriginatorOrg: mitprod.onmicrosoft.com

[tompollard]

This is definitely an issue to look into. Many of the emails are also going to my spam folder.

[tompollard]

A few thoughts on improving our emails:

1. PTR records

Are our PTR records set up correctly? A reverse DNS lookup on our IP returns physionet.mit.edu. This is not the domain used in our sender address. Is that a problem?

dig -x 18.13.52.205

;; ANSWER SECTION:
205.52.13.18.in-addr.arpa. 1800 IN  PTR physionet.mit.edu.

2. DomainKeys Identified Mail (DKIM)

I assume that DKIM has not been set up, but it seems like a good idea. The process involves (1) creating public/private keypairs (2) adding a couple of TXT records to our DNS, one which references the public key (3) signing the emails with the key.

3. SPF / SenderID record in your DNS

This seems the most straightforward of the three steps (and the least beneficial). It involves adding our sender IP/s to another TXT record in the DNS.

[tompollard]

@Lucas-Mc @bemoody this seems to be a fairly major issue, because from a couple of recent messages it seems that some networks are completely blocking our emails (they don't even arrive in spam folders). Has anyone had a chance to look into it? If not, I'll take a look into the steps above today.

[bemoody]

I set up DKIM last week. Look at recent messages, you should be able to verify them (e.g. using opendkim-testmsg.) SPF was done ages ago.

I wouldn't assume something's wrong with the mail server just because people complain. The system is fragile in a lot of ways that are not related to email per se. Have you verified that the messages were actually sent in the last few days and that they were accepted (or rejected) by the destination server?

[tompollard]

Thanks for setting up the DKIM, and glad to hear that SPF is already in existence! Do you have any thoughts on the first point, around PTR records (i.e. does it matter that the reverse DNS lookup returns physionet.mit.edu when our sender address is physionet.org?).

I wouldn't assume something's wrong with the mail server just because people complain. The system is fragile in a lot of ways that are not related to email per se. Have you verified that the messages were actually sent in the last few days and that they were accepted (or rejected) by the destination server?

All I know is that there are reports from ~3 or 4 users saying that (1) registering with an institution email address was not possible because the email wasn't received (2) switching to a different address immediately solved the problem.

I haven't investigated why this is happening (e.g. it is possible that the emails were just not sent at all), but it seems likely that something may be preventing delivery. I'm happy to explore, but if so it would be helpful to chat to understand more about what I should be looking out for (unfortunately I'm not able to get onto jabber at the moment, but I'll try to fix that!).

[bemoody]

On 8/19/20, Tom Pollard notifications@github.com wrote:

Thanks for setting up the DKIM, and glad to hear that SPF is already in existence! Do you have any thoughts on the first point, around PTR records (i.e. does it matter that the reverse DNS lookup returns physionet.mit.edu when our sender address is physionet.org?).

Certainly the conventional wisdom was always that it didn't matter. It's not at all uncommon to have many domains' outgoing mail routed through a single IP address, and obviously one address can't be mapped to multiple hostnames at once. Even big senders don't follow that rule: if you send mail "from" gmail.com, the machine that actually sends it is something-or-other.google.com.

The only sensible way to check if an IP address is allowed to send mail for a particular domain would be... SPF, or something very much like it. But nothing about email is sensible nowadays, so who knows?

BTW, you're looking at the wrong IP address. Outgoing mail comes from 128.30.29.50, not 18.13.52.205. It doesn't matter what 18.13.52.205 maps to.

All I know is that there are reports from ~3 or 4 users saying that (1) registering with an institution email address was not possible because the email wasn't received (2) switching to a different address immediately solved the problem.

A good place to start would be to make a list of domains that are failing. Of course, check whether the messages are being accepted or rejected. See if there are any patterns, and try to find out if there's anyone on the other side who's both willing and able to debug the problem (a tall order, I'm sure!)

[tompollard]

BTW, you're looking at the wrong IP address. Outgoing mail comes from 128.30.29.50, not 18.13.52.205. It doesn't matter what 18.13.52.205 maps to.

Got it, so it looks like heimdallr.csail.mit.edu which is even further away!

A good place to start would be to make a list of domains that are failing. Of course, check whether the messages are being accepted or rejected. See if there are any patterns, and try to find out if there's anyone on the other side who's both willing and able to debug the problem (a tall order, I'm sure!)

Good suggestion, I will collate a list of domains and then we can work out next steps.

[tompollard]

After looking into this (and thanks to a related bug for one of our users), I think I understand what is happening (and @bemoody, you were correct to question my assumption that the problem was emails being filtered!).

In all cases where emails were reported not to have been received, the user was attempting to register. After registration, later emails were fine. What these users had in common was that they did not have an associated email in the system.

The reasons for this can be discussed in a separate issue, but either (1) the associated email object was not created or (2) the associated email object was deleted in a cron task.

The fix for this is to follow the steps in the AssociatedEmail.create_associated_email method to create a new AssociatedEmail for the user.

It sounds like we are doing what we can to ensure emails are delivered to our users, so I think this issue can be closed.