kelson42 commented 1 year ago

We would like to inform you that our scan has identified one problem with your SPF record. These issue prevent SPF from working correctly and as a result, emails sent from your domain can be forged. Specifically, we have found the following issue:

Too many DNS lookups

The SPF record requires more than 10 DNS queries and is therefore invalid [3]. Currently, the SPF record triggers 34 requests. We recommend replacing some domain names with IP addresses. If you are uncertain whether your SPF entry is correct, there are online tools that can help you, such as:

kelson42 commented 1 year ago

I had briefly a look to:

rgaudin commented 1 year ago

That's for kiwix.org

https://mxtoolbox.com/SuperTool.aspx?action=spf%3akiwix.org&run=toolpage

benoit74 commented 10 months ago

The main problem seems to be that we are using many services to send emails on behalf of our domain, and each services is using many sub-services / IPs:

gaggle.email
email.freshdesk.com
cyon.ch
_mailcust.gandi.net

Is there any service we do not use anymore? Or any we could switch to use gandi SMTP servers (with proprer authentication of course...)?

I digged a bit into the SPF records (with https://gist.github.com/benoit74/35cb8b01d3a6aa4a91ad985a5de9ed57) and this is what I found:

kiwix.org: 11 instructions, among which 4 includes
  gaggle.email: 5 instructions, among which 3 includes
    _spf.firebasemail.com: 4 instructions, among which 2 includes
      _spf.google.com: 5 instructions, among which 3 includes
        _netblocks3.google.com: 12 instructions, among which 0 includes
        _netblocks2.google.com: 8 instructions, among which 0 includes
        _netblocks.google.com: 13 instructions, among which 0 includes
      sendgrid.net: 12 instructions, among which 1 includes
        ab.sendgrid.net: 6 instructions, among which 0 includes
    helpscoutemail.com: 8 instructions, among which 0 includes
    _spf.gaggle.email: 14 instructions, among which 1 includes
      amazonses.com: 13 instructions, among which 0 includes
  email.freshdesk.com: 7 instructions, among which 5 includes
    fdspfaus.freshemail.io: 12 instructions, among which 0 includes
    fdspfind.freshemail.io: 12 instructions, among which 0 includes
    fdspfeuc.freshemail.io: 13 instructions, among which 0 includes
    fdspfus.freshemail.io: 13 instructions, among which 1 includes
      fdspfus2.freshemail.io: 6 instructions, among which 0 includes
    sendgrid.net: 12 instructions, among which 1 includes
      ab.sendgrid.net: 6 instructions, among which 0 includes
  cyon.ch: 9 instructions, among which 4 includes
    _spf.mailrelay.rrpproxy.net: 3 instructions, among which 1 includes
      spf.key-systems.net: 5 instructions, among which 0 includes
    servers.mcsv.net: 5 instructions, among which 0 includes
    helpscoutemail.com: 8 instructions, among which 0 includes
    spf.patchman.co: 3 instructions, among which 0 includes
  _mailcust.gandi.net: 3 instructions, among which 1 includes
    _nblcust.gandi.net: 10 instructions, among which 0 includes

gaggle.email seems to be pretty deeply nested / using many subservices.

cyon.ch and freshdesk are also not very good.

rgaudin commented 10 months ago

We use them all ; I cleaned it up already in May

rgaudin commented 10 months ago

I think there are at least those options:

Pay a fee to a flattening service. That's the easiest. We pay (autofps is $324/y), change the SPF once initially and that's it.
Setup a dynamic solution like expurgate which adds a SPOF for emails on our infrastructure as with this setup, every recipient's email provider triggers at least one DNS call to this service. We'd also need to make the initial SPF record change and we have to maintain it (keep it alive, mostly).
Bake something ourselves. There are probably pieces of code doing this but it's relatively simple anyway: recusrively loop over our sources, query/parse SPF and get IPs (de-duping). Could run on GH CI and either update record (I believe GANDI has an API) or just fail CI so we manually update records (I imagine those 20 names dont change IPs every week). Could be an intern/student task to build.

From what I understand, we don't need Gaggle in our SPF record. We do use it but it doesn't send from an @kiwix.org address. Gaggle is a mailing-list/groups service and we have a handful of those (@Popolechien please share Gaggle credentials btw). We thus has address list shortcut@kiwix.org that gets redirected into an @gaggle.email address and Gaggle sends all email with a From of his own.

I've thus removed it from the SPF record. We're still way above the limit (at 20 instead of 10).

I've also reconfigured (via a Plugin) WP to send emails via Mailgun and I've also configured the MailPoet Plugin to send via Gandi SMTP. Those changes allows us to remove the cyon include from the SPF record.

We're now at 12 records instead of 10 (arg!). The remaining ones are: Gandi (for SMTP), Mailgun and Freshdesk.

Freshdesk is our helpdesk and we can't spare it but… I just checked and we can configure it to use Gandi's IMAP and SMTP instead of the current setup (SPF to use From: xxx@kiwix.org and incoming via a redirect to xxx@kiwix.freshdesk.com).

Given how nested freshdesk records are, it's clear we can stay within 10 once we remove it. I haven't done it (not my call) but it's an easy switch.

For future reference, here is an exhaustive list (best effort) of our services sending email:

Service	From	Via
Freshdesk	`contact@kiwix.org`	FreshDesk	Helpdesk responses
Regular Emails	`*@kiwix.org`	Gandi SMTP
Wordpress MailPoet Plugin	`hello@kiwix.org`	Gandi SMTP	Newsletter
Wordpress	`stephane@kiwix.org`	Mailgun	Password reset and notifications
Cardshop-scheduler	`hotspot@kiwix.org`	Mailgun	Startup notif, Order notification emails
Cardshop-manager	`hotspot@kiwix.org`	Mailgun	Password reset
Matomo	-	-	Not configured but could password reset and send reports
Kiwix Wiki	`wiki@wiki.kiwix.org`	Mailgun	Password reset & notifications
openZIM Wiki	`wiki@wiki.openzim.org`	Mailgun	Password reset & notifications
Zimfarm API	`info@farm.openzim.org`	Mailgun	Notifications (none configured ATM)
youzim.it Zimfarm API	-	-	Could send notifications via Mailgun (not configured)
youzim.it frontend	`info@youzim.it`	Mailgun	ZIM request status update (created, completed)

rgaudin commented 9 months ago

Freshdesk has been migrated to a new IMAP BAL. Its include statement has been removed and now our SPF is valid 🎉

Check for yourself

kiwix / operations

SPF record is too long #87

Too many DNS lookups