Bug: Our emails to Gmail going to spam

benhylau commented 3 years ago

Incident Description

What happened?

In #418 @patcon described forwarded emails for his Gmail end up in spam ~~or dropped altogeter~~ [edit: no drops, just some starting to be spam-filed --patcon]
When signing up to our listservs, the verification email goes to spam of Gmail targets

Expectated Behaviour

What did you expect to happen instead?

Gmail to stop hating on us

Priority

How severe and urgent do you think this is?

Severe and urgent because #418 started happening recently for @patcon, this did not happen before, so if there is a new reason for Google to hate us, our "Gmail reputation" will get progressively worse, which affects all our email communications moving forward, including client communication and listservs.

Hypothesis

What do you think caused this?

We disabled basic spam protection (aka. SUBSCRIBE_FORM_SECRET) to allow for embedded web forms. There is no way to embed a web form with that enabled. @ASoTNetworks has confirmed that our lists are getting hit with spammer bots, which would cause large amounts of verification emails be sent to subscription addresses, many of them are probably Gmail. (Sidenote: one of those bounced, that's one reason why we took notice.) While we are not certain, this is likely the cause of our Gmail reputation sinking.

Proposed Solution

If you have a proposal on how to address this let us know!

1. Re-enable `SUBSCRIBE_FORM_SECRET`

This kills embedded web forms. We need to notice COMPOST and Toronto Mesh to remove the embeds in their websites. This, however, is not even a strong protection in the first place, as some bots still get through, but it's an easy one at the expense of having to re-expose ugly mailman UI.

2. CAPTCHA

There are patches to modify mailman to support this, but based on reCAPTCHA from Google. We probably want an implementation that is not using Google's as being independent of Google services was kind of the point of hosting our lists in the first place.

Secondly, there are concerns with efficacy and accessibility Some think it's basically pointless, and everyone hates it, myself included.

3. Self-implemented bot check

For example: "What is 1 + 1?" Easy to implement, obviously easy to bypass for a determined spammer. But much less annoying than full CAPTCHA for the human. Probably more accessible.

Question: can we implement this in the main mailman interface as well?

4. Honeypot

@YurkoWasHere mentioned this as well, the "hidden field" (aka. honeypot) would require no human intervention. I assume we'd want to hide the text field element visually with js.

See: http://mailman.9.n7.nabble.com/CAPTCHA-support-tp44131p44164.html

Same question: can we implement this in the main mailman interface as well?

I personally prefer 3 and 4, and we have to consider both the spam on the mailman interface and from embedded web forms. It's possible we will need to do 1 regardless.

In the meantime, we should get folks to subscribe to our lists with their Gmail, then login to a Gmail app, and Report not spam to tell Google to stop hating on us.

Screen Shot 2021-03-06 at 12 53 24 PM

benhylau commented 3 years ago

I started receiving these list bounce notices every couple days, each time in batches, since Feb 17. They all went to my spam, but they are continuing to this day.

Screen Shot 2021-03-06 at 3 29 22 PM

patcon commented 3 years ago

Thanks @benhylau + infra.

Alternative: start hosting lists on another of our toy domains so we don't get hit on our client-facing infra if/when lists are abused. i.e. Learn to improve list hosting under less punishing circumstances

patcon commented 3 years ago

@YurkoWasHere's comment applies here too, since it seems to negate current hypothesis:

listserv is a different mail server and different ip address, and probably does not have any affect on the mail cow instance -- https://github.com/hyphacoop/organizing/issues/418#issuecomment-792166309

benhylau commented 3 years ago

@YurkoWasHere which brings me to my other question... is reputation by IP or domain? Or both?

benhylau commented 3 years ago

Can @garrying @YurkoWasHere @ASoTNetworks pick this up? I think it is quite urgent as we are getting spammed constantly.

uditvira commented 3 years ago

Last week a critical email to udit@hypha.coop got marked as spam by gmail. The individual followed up so we caught it. I normally use apple mail, but I recently upgraded to Big Sur and I'm in the middle of migrating all my accounts, and have been using gmail as a stopgap solution.

Why is it that 90% of emails make it through gmail's spam filters but a minority don't?
Can I get infra wg to give me status on this? If we don't support gmail, can we be definitive about that?

ASoTNetworks commented 3 years ago

@uditvira was the emails sent to a working group email or directly to you that get marked as spam?

ASoTNetworks commented 3 years ago

In the weekend we turned on the token option on Mailman to prevent spam being sent out by mass registration by bots. Hopefully this will help with our spam saturation.

uditvira commented 3 years ago

@uditvira was the emails sent to a working group email or directly to you that get marked as spam?

Directly to udit@hypha.coop

ASoTNetworks commented 3 years ago

It looks like our SPF record for hypha.coop wasn't resolving I have fixed that issue and lets see if it helps.

garrying commented 3 years ago

From March 23, 2021, infra-wg meet:

Issue 1: Google screwing up and marking email as spam

el has updated SPF record on March 23rd
infra-wg will check back after 24-48 hours to see if this resolves the issue
Next steps:
- Ask members to help in seeing if critical emails are still being marked as spam past March 25th
- Aiming to close issue March 26

Issue 2: Hypha Listserv is spamming

el has turned on Mailman secret token on the web form on March ~20th (weekend)
- This means we can't embed the form for now
- So far there has not been spam coming from the mailman server
Similar to Issue 1: We need to confirm with Google if mail from @hypha.coop is still being marked as spam
Next steps:
- Resolve known embeds of the form:
- el will ping bl to resolve issue with IPFS and one.compost.digital
- PR for removing the form (tomesh): https://github.com/tomeshnet/tomesh.net/pull/305
- infra-wg will checkback in 24-48 hours (within the weekend)
- Aiming to close issue March 26

ASoTNetworks commented 3 years ago

It looks like something keeps removing the SPF record from hypha.coop.

@hyphacoop/infrastructure-wg is investigating the cause.

patcon commented 3 years ago

If we don't support gmail, can we be definitive about that?

My sense is that issues would still remain with externals getting our email marked as spam. My assumption is that "don't support gmail" is not an option, as that's masking external issues, right? Please do confirm that this is not in the cards @hyphacoop/infrastructure-wg 🙏

Ask members to help in seeing if critical emails are still being marked as spam

My current gmail filter work-around skips spam for any emails to/from *@hyph.coop, so I won't notice any improvements here. Would it be helpful for me to disable that work-around for now, and just watch my spam closely?

ASoTNetworks commented 3 years ago

@hyphacoop/infrastructure-wg found the cause of the service removing SPF records and added the record back.

I now see spf=pass in the emails forwarded to Gmail.

@patcon if you want to try it go ahead.

garrying commented 3 years ago

@patcon I think it would be helpful if you're okay with checking spam over a period of time until you feel the behaviour is conclusive.

I'm closing this issue for now. Feel free to reopen this issue or create a new one.

hyphacoop / organizing