search

GSA / data.gov

Main repository for the data.gov service
https://data.gov
Other
667 stars 104 forks source link

Harvest notification emails are not going through to agency contacts #507

Closed adborden closed 4 years ago

adborden commented 6 years ago

@hkdctol and other GSA contacts are receiving the emails, but agency contacts are not. Seems to have been happening for the past ~2 weeks.

adborden commented 6 years ago

@hkdctol sent me a copy of one of the messages. I didn't spot anything unusual with it. It passes SPF but is not signed with DKIM. Everything looks in order. Are the messages going to spam, or just not being delivered?

adborden commented 6 years ago

The only errors in the log are related to empty recipient addresses and they seem rare. So if there is an issue, it's between gsa's email server and the agency's email server. We could reach out to GSA to see if they've been seeing any issues sending our mail.

hkdctol commented 6 years ago

@adborden ok - let me check with agency contacts and then follow up with GSA email contacts.

hkdctol commented 5 years ago

This seems resolved--will close for now and reopen if we run into any agencies that are not receiving harvest reports through email.

adborden commented 4 years ago

Re-opening since we are getting reports from agencies that they are not receiving harvest notifications.

adborden commented 4 years ago

We updated our DMARC record to include a reporting address so that we can see and analyze delivery reports.

adborden commented 4 years ago

USAID says they haven't received an email since 3/25. Harvest jobs have been run everyday since then and I do see the email address in the mail.log on catalog-harvester1p

Mar 29 16:04:45 catalog-harvester1p postfix/smtp[24423]: 9CEC960584: to=<mdaniels@usaid.gov>, relay=smtp.gsa.gov[159.142.1.100]:25, delay=0.79, delays=0.01/0.01/0.62/0.16, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as DDAB721DE4)

Not seeing any errors in Agari, either.

mogul commented 4 years ago

@adborden is following up with USAID in a mail thread "Fwd: Email being marked as SPAM from data.gov".

adborden commented 4 years ago

I've added our USAID contact to the harvest job as a test and sent a test message directly from the harvest host (using mail). Moving to blocked until we hear back.

mogul commented 4 years ago

Gonna give this until Monday to see if they understand and can supply what we're asking for, then will follow up.

adborden commented 4 years ago

Heard from USAID, they are receiving emails but they are going to spam. Given our DMARC policy is reject, it seems unrelated to DMARC. They are still doing some investigation on their side, no action for us at this point.

mogul commented 4 years ago

@adborden will ping to see if there's any movement on the USAID side. However, we're going to close this issue since spam detection tuning on the receiving end is totally out of our control. If they come back with something specific we need to change, then we'll open a new issue.

hkdctol commented 4 years ago

Reopening. Heard from Energy, DOJ, and NEH yesterday that they have not received harvest reports since May 9. Same date for all. Checked harvest results from admin UI and the harvest reports have been going through. USAID also still not getting harvest emails.

Harvest report emails to internal gsa addresses being received as usual.

hkdctol commented 4 years ago

I don't think we've done any work since this ticket was reopened, but hearing from USAID and PBGC that daily harvest emails that they have not been getting for a while started coming through today. Waiting to hear back from other agencies.

hkdctol commented 4 years ago

DOJ got emails starting today too. Not sure we've ever really investigated anything specific. Maybe next step is asking GSA email team if something changed, since in this instance we have some consistent dates on when the problem started/stopped.

hkdctol commented 4 years ago

Seems like all the agencies are getting the emails. Reached out to a GSA email team member we consulted in the past to see if there's any explanation on the date range, May 9 - June 9.

hkdctol commented 4 years ago

All the agencies are getting the emails. We got some additional information from GSA email team, which we will relay to USAID.

hkdctol commented 4 years ago

Re-opening as I have multiple reports from agencies that the last harvest email received was September 7.

adborden commented 4 years ago

FYI, I've confirmed that we're handing off emails properly to the GSA SMTP relay. It looks like the emails are going through to GSA addresses, I received the DOJ report:

Oct  1 17:05:19 catalog-harvester1p postfix/smtpd[13327]: connect from localhost[127.0.0.1]
Oct  1 17:05:19 catalog-harvester1p postfix/smtpd[13327]: A4CB060344: client=localhost[127.0.0.1]
Oct  1 17:05:19 catalog-harvester1p postfix/cleanup[13330]: A4CB060344: message-id=<20201001170519.A4CB060344@catalog-harvester1p.prod-ocsit.bsp.gsa.gov>
Oct  1 17:05:19 catalog-harvester1p postfix/qmgr[5189]: A4CB060344: from=<no-reply@data.gov>, size=1771, nrcpt=4 (queue active)
Oct  1 17:05:19 catalog-harvester1p postfix/smtpd[13327]: disconnect from localhost[127.0.0.1]
Oct  1 17:05:20 catalog-harvester1p postfix/smtp[13331]: A4CB060344: to=<aaron.borden@gsa.gov>, relay=smtp.gsa.gov[159.142.67.242]:25, delay=0.64, delays=0.01/0.01/0.47/0.16, dsn
=2.0.0, status=sent (250 2.0.0 Ok: queued as 582BC29825)
Oct  1 17:05:20 catalog-harvester1p postfix/smtp[13331]: A4CB060344: to=<crystal.carter@gsa.gov>, relay=smtp.gsa.gov[159.142.67.242]:25, delay=0.64, delays=0.01/0.01/0.47/0.16, d
sn=2.0.0, status=sent (250 2.0.0 Ok: queued as 582BC29825)
Oct  1 17:05:20 catalog-harvester1p postfix/smtp[13331]: A4CB060344: to=<jake.bishopgreen@usdoj.gov>, relay=smtp.gsa.gov[159.142.67.242]:25, delay=0.64, delays=0.01/0.01/0.47/0.1
6, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 582BC29825)
Oct  1 17:05:20 catalog-harvester1p postfix/smtp[13331]: A4CB060344: to=<monique.bourque@usdoj.gov>, relay=smtp.gsa.gov[159.142.67.242]:25, delay=0.64, delays=0.01/0.01/0.47/0.16
, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 582BC29825)
Oct  1 17:05:20 catalog-harvester1p postfix/qmgr[5189]: A4CB060344: removed

We should open an incident with the GSA SMTP team.

adborden commented 4 years ago

I submitted a ticket with the email source.

adborden commented 4 years ago

Another report https://github.com/GSA/data.gov/issues/907

hkdctol commented 4 years ago

Multiple agencies reporting they started getting the harvest report emails last night.

mogul commented 4 years ago

Leaving open until we see explicit confirmation of a change made on the GSA side that will prevent this improvement from regressing.

adborden commented 4 years ago

Here's the latest from GSA:

It looks like the Google SMTP server is changing the MAIL FROM to postmaster@gsa.gov for non-gsa.gov recipients, which is definitely wrong.

Using debug_peer_list = smtp.gsa.gov in the Postfix configuration, I was able to get a trace of the SMTP conversation on staging. This trace shows we are correctly setting the MAIL FROM as no-reply@data.gov on the handoff to GSA's SMTP server.

FWIW, I think the postmaster@gsa.gov issue is not new... we saw it with USAID but focused on a separate issue which seemed to resolve the delivery problems. Not sure if this is intermittent, or unrelated, or overlooked.

adborden commented 4 years ago

No response yet from GSA.

In the meantime, agencies are reporting that they are receiving harvest reports without any change on our side (or presumably GSA's).

Yet despite deliveries, I'm still seeing the From being replaced with postmaster@gsa.gov for non-gsa.gov recipients. Here you can see Google's mail servers connecting to mail.a14n.net, and the message being rejected with from=<postmaster@gsa.gov>:

Oct 15 00:49:56 a14n postfix/postscreen[7230]: CONNECT from [209.85.221.226]:33639 to [64.227.84.77]:25
Oct 15 00:50:02 a14n postfix/postscreen[7230]: PASS NEW [209.85.221.226]:33639
Oct 15 00:50:02 a14n postfix/smtpd[7231]: connect from mail-vk1-f226.google.com[209.85.221.226]
Oct 15 00:50:03 a14n postfix/smtpd[7231]: 4A5D313B0D9: client=mail-vk1-f226.google.com[209.85.221.226]
Oct 15 00:50:03 a14n postfix/cleanup[7257]: 4A5D313B0D9: message-id=<20201015004954.4E213BBE15@catalogharvester1d.dev-ocsit.bsp.gsa.gov>
Oct 15 00:50:06 a14n postfix/cleanup[7257]: 4A5D313B0D9: milter-reject: END-OF-MESSAGE from mail-vk1-f226.google.com[209.85.221.226]: 5.7.1 Spam message rejected; from=<postmaster@gsa.gov> to=<adborden@a14n.net> proto=ESMTP helo=<mail-vk1-f226.google.com>
Oct 15 00:50:06 a14n postfix/smtpd[7231]: disconnect from mail-vk1-f226.google.com[209.85.221.226] ehlo=2 starttls=1 mail=1 rcpt=1 data=0/1 quit=1 commands=6/7

Even with deliveries appearing resolved, we should press GSA on resolving the incorrect From address.

mogul commented 4 years ago

Leaving this Blocked while we wait on an explicit response from GSA IT.

adborden commented 4 years ago

The replacement of the From address to postmaster@gsa.gov, is part of an anti-spam feature of Google. Since data.gov is not registered as a GSA Google domain. This doesn't completely explain things, because while this was implemented, some mail is still going through.

That means that GSA SMTP service now requires any non-gsa.gov domains to be registered with GSA Google.

Long-term: we want to move away from the GSA SMTP service since it won't be able to support the cloud.gov use case where we are outside of the GSA network.

Short-term: GSA will add data.gsa.gov as a domain and we'll be able to send from no-reply@data.gsa.gov

adborden commented 4 years ago

GSA has opened a ticket for the new DNS entries.

adborden commented 4 years ago

The new sending domain is ready. I just tested it on datagov-jump2d and it seems to work. PR incoming...

jbrown-xentity commented 4 years ago

PR verified deployed on latest ckan production.ini file... smtp.mail_from = no-reploy@data.gsa.gov. Do not have a valid example in the wild.

adborden commented 4 years ago

Trying to keep a list of emails we send so that we know where the FROM address needs to be updated in the future