Set proper DMARC (DKIM & SPF) records for backdropcms.org

[alanmels]

While troubleshooting why I could not get my password from backdropcms.org, I noticed the domain name didn't not have DMARC set at all (checked on https://mxtoolbox.com/SuperTool.aspx?action=mx%3abackdropcms.org&run=toolpage):

Please note that while most of the outgoing messages from backdropcms.org are successfully delivered, because in order not to miss any incoming mail most of the mailservers still have loose rules, at the same time some of the modern mailservers will strictly reject the same messages. So ideally DMARC records should be set and look like:

Additional information on DMARC: https://dmarc.org, https://en.wikipedia.org/wiki/DMARC

[alanmels]

More somehow related information to consider...

The whois information for backdropcms.org shows the domain name is registered through Namecheap.com and at the same time it uses the following nameservers:

Name Server: dns1.registrar-servers.com 
Name Server: dns2.registrar-servers.com

Now, if you check the whois information of the domain name registrar-servers.com, then you will see it is managed by Enom.com, which in turn belongs to Tucows. The controversy here I see is that there was legal dispute between Namecheap and Enom (please read https://domainnamewire.com/2017/09/01/namecheap-sues-enom-tucows-demands-transfer-4-million-domains, https://domainnamewire.com/2018/11/08/namecheap-and-tucows-settle-legal-dispute) and I see potential problems if any domain is registered through one registrar and manages its DNS through another registrar and there were legal issues between the two, even already set ones.

On the other hand, there is probably no reason to worry if it has been working as expected up until today. However, my personal preference is always to avoid services of companies registered in Panama (Namecheap), because all those off-shore companies smell fishy to me. The ideal registrar at the moment would be Cloudflare, which also helps to save some money (see their pricing on https://www.cloudflare.com/products/registrar). Also entrusting DNS for the domain name to Cloudflare would ensure additional security, stability and high-availability.

[klonos]

cc'ing @quicksketch and @larsdesigns

[oadaeh]

I also do not see any DKIM records set, which needs to be done before fixing DMARC.

[alanmels]

Unfortunately, I did not receive any mail after getting registered on the main website at https://backdropcms.org. Attempt to retrieve a new password on https://backdropcms.org/user/password also doesn't send anything.

Guys, could anyone try to request a new password and see if you get any message from https://backdropcms.org?

[oadaeh]

@alanmels I received the message, but it went to spam. Maybe yours did too?

[alanmels]

@alanmels I received the message, but it went to spam. Maybe yours did too?

@oadaeh Thanks for the hint. After registering a test account using another mail service, I finally see where the problem is: messages sent from li505-83.members.linode.com, but using the no-reply@backdropcms.org FROM address were failing because of our strict DMARC requirements for incoming mail. I am afraid we have to loosen them up until/if the reported problem is addressed.

[jenlampton]

I wonder if our DMARC thing needs to be updated since we changed servers? @quicksketch

[oadaeh]

@jenlampton @quicksketch maybe we need to change the reverse DNS address? https://www.linode.com/docs/networking/dns/configure-your-linode-for-reverse-dns/

[larsdesigns]

I went ahead and updated the reverse dns records for both of the hosts within linode:

1. www2.backdropcms.org is now backdropcms.org
2. Created rDNS for ci.backdropcms.org.

I am not sure that DMARC has actually been adopted as a standard but it is probably a good thing to do. We should start by creating an SPF record and setting up DKIM is helpful.

As far as I know, I do not have access to the DNS Zones to make those changes but I will look into it.

[jenlampton]

Thank you @larsdesigns !

[alanmels]

Unfortunately, the https://mxtoolbox.com/SuperTool.aspx?action=mx%3abackdropcms.org&run=toolpage still shows the first screenshot in https://github.com/backdrop-ops/backdropcms.org/issues/607#issue-522720712.

@larsdesigns, let me know if you need additional hands on how to get everything look like https://mxtoolbox.com/SuperTool.aspx?action=mx%3aaltagrade.com&run=toolpage

[larsdesigns]

@alanmels, It is not that I need "additional hands". As I stated before, I do not have access to the DNS for backdropcms.org. So I will have to refer the next steps to @quicksketch and/or @jenlampton.

[larsdesigns]

Note: The PTR records may need to updated also within the zone.

[larsdesigns]

Another option would be to use an SMTP relay service such as Amazon's SES. Which could be advantageous for sending more e-mails per day than what Linode allows. I don't remember their limit off-hand but I think was at 1000. Perhaps newsletter, notifications, etc could push past that limit sooner than we may except.

[jenlampton]

@larsdesigns If we do choose to use a dedicated smtp provider for b.org, we'll probably choose to use the same one as we do for newsletters, so that would be Sendgrid, I think. But let's push that decision until CiviCRM is set up on b.org.

Also for the record, I do not have access to DNS either, I believe @quicksketch is the only one with access. We discussed this week setting up a new dedicated account for Backdrop and putting the creds into the shared Backdrop 1Password account, but I don't think it has been done yet.

[quicksketch]

@larsdesigns I can't easily give you DNS access yet, I need to separate Backdrop's domains into a new account. However, I can set up new DKIM keys. I've done so, making a TXT record for

www2._domainkey.backdropcms.org
TXT
v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCvsejE1HXhPk2e7lq++CARuc2MLeB5SwJaKINqgqnRrB6zUuDeXPVnY+sFTctFd1Z2ZmbZhFmFXyTgrqaQSRupecSFtiLzJlodZ/iPnkQgZ2Y+8BGQ8ccUeeveztqBDA4wQI9yHxUclbtwFRHMOoVLrh5/IcoT2oPHlT8kjOtSFQIDAQAB

I'll send you the private key separately. Could you use that to configure postfix to sign outgoing emails?

[larsdesigns]

@quicksketch, I was having trouble using the keys that you provided. I went ahead and created keys and followed the configuration process as documented at: https://www.linode.com/docs/email/postfix/configure-spf-and-dkim-in-postfix-on-debian-9/.

There are some small differences when it comes to the socket location though and I only completed the DKIM configuration and not the SPF record configuration. I will continue with that.

1. [x] Please adjust the text record that you created with the following content:

v=DKIM1; h=sha256; k=rsa; s=email; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzORGOnrycJPDF5eSbIZoM1WqEL1gGaYbsSSykPZTD15pk5u8pG3SKvNLAwxWHbe0nIuWucO+fmNp7J2sGrVduxF68wjemrbbJytk0LZIcUckBc0c7Pn85d8Bb25NYLXmCLOVKpwxyj7khTWoqh0UlNYYqlEFgBnwVzdCfH5n3crZ/iC/UAaJwSDZLQg5zCQ4X8q6QD5HDxGbeY

I am receiving password reset e-mails in my gmail account. We are on the right track.

[larsdesigns]

@quicksketch,

1. Finished the configuration for SPF within postfix. The existing TXT SPF record looks acceptable when dig returned it for me.
2. Tested by sending a password reset e-mail.
3. Let's test again when the DKIM TXT record is updated by using this command: opendkim-testkey -d backdropcms.org -s 201911

[alanmels]

Unfortunately, none of the entries on https://mxtoolbox.com/SuperTool.aspx?action=mx%3abackdropcms.org&run=toolpage has progressed.

[larsdesigns]

@alanmels is not correct when he states "none of the entries as progressed."

All of the entries have been corrected as reported by mxtoolbox.com except for the creation of the dmarc record.

The information from mxtoolbox.com:

If you are encountering this error of No DMARC Record found, this means that your domain does not have a published DMARC record. DMARC Records are published via DNS as a text(TXT) record. They will let receiving servers know what they should do with non-aligned email received from your domain.

I do not have access to create the dmarc record. If we can finish making the DNS records, then folks that only accept encrypted e-mail will be able to receive e-mail.

@quicksketch, Just to double check, did you created the DKIM TXT record? I am only seeing the SPF record when I dig from google dns. I thought I remember you completed that.

Should we move forward on creating a dmarc record also?

[alanmels]

@alanmels is not correct when he states "none of the entries as progressed."

Are we looking at two different pages?

The screenshot I attached in the first post was taken on on Nov 14, 2019. Now compare with today's state:

@larsdesigns, pardon me, but can you tell me what has changed in that picture?

I do understand that you are doing something in background, however anyone can clearly see that from the perspective of public services like https://mxtoolbox.com nothing changed. And please do not take this personal, I do understand that you, guys, are trying to fix this, but please be objective - the effect of the changes you are making needs to be reflected publicly.

[larsdesigns]

Like I said, everything has been corrected except for the dmarc record being created. Here are some highlights as found in the issue history:

1. SPF record was created.
2. DKIM encryption has been configured.
3. PTR record has been updated.

To say there has been no progress made may not seem very considerate to the volunteers who are working to accommodate your reported e-mail problem.

Most of us are probably using Gmail or some other e-mail as a service product where dmarc is not required. While I appreciate having additional security, we will get there eventually.

[alanmels]

@larsdesigns, I do think you are not fair here. Just read what I had began today with:

Unfortunately, none of the entries on https://mxtoolbox.com/SuperTool.aspx?action=mx%3abackdropcms.org&run=toolpage has progressed.

and nothing else! I didn't say anything about what you might have done or not behind the curtains. My statement above clearly says nothing has changed on the provided link, which is true.

Anyway, I don't think it's productive to be exchanging posts like this. Could we just go ahead and start making the picture on the screenshot finally changed?

[alanmels]

your reported e-mail problem

@larsdesigns, stay assured its not my e-mail problem, but of backdropcms.org. As I tried to explain in the first post some of modern servers will reject notifications coming from backdropcms.org.

Most of us are probably using Gmail or some other e-mail as a service product where dmarc is not required.

Please read https://github.com/backdrop-ops/backdropcms.org/issues/608 to see what Gmail thinks of notification that came from backdropcms.org. Just look at https://mxtoolbox.com/SuperTool.aspx?action=mx%3abackdropcms.org&run=toolpage to understand that the problem is not on my or anyone else's end.

Please don't make it sound like I need this most as I am trying here to push things that would only benefit to Backdrop, the main website of the project in this case, for God's sake.

P.S. If our customers reported this kind of issue back on November 14th and we couldn't fix it all the way until January 22, then I would feel really ashamed. And we are talking about the main website of Backdrop project, which should have been setup perfectly. Really sad, that we can not even ping this kind of issues just to remind they are not resolved without getting punished like this.

[larsdesigns]

@alanmels, I do not feel ashamed at all. I am a volunteer and this issue is not assigned to me. I am unable to make the changes you are requesting because I do not have access to DNS as stated earlier.

I am really happy for you that your customers promptly receive dmarc configured.

I am not aware of anyone getting punished and I appreciate you bringing up this issue again.

[alanmels]

@larsdesigns Look Justin, the subject matter is really not about you, who you are, how you are volunteering and what you've done or not. I do appreciate everything you are doing for the project, but let's concentrate on resolving the real issues.

If you do not have access to DNS, then someone who has access have to start acting. It's not really about you. Sorry if my pinging this issue up again somehow has hurt your feelings, but you must agree that it needs further attention. Please don't take things personal, we are all trying to make Backdrop better, in this case the main website of the project work better.

[larsdesigns]

@alanmels, I am really not ashamed and my feeling are not hurt. I do not appreciate you trying to put that spin on the thread.

I recommend trying to not make your communication personally targeted. Please do not assume how someone feels to control the narrative to make it look like someone should be ashamed or someone has hurt feelings. This kind of language should be left out of the conversation. I never suggested that I was feeling one way or the other. I was only focussing on the subject matter of the thread.

Quotes from your comments that really are not appropriate:

P.S. If our customers reported this kind of issue back on November 14th and we couldn't fix it all the way until January 22, then I would feel really ashamed. And we are talking about the main website of Backdrop project, which should have been setup perfectly. Really sad, that we can not even ping this kind of issues just to remind they are not resolved without getting punished like this.

Your customers, your business, suggesting that someone should be shamed, etc, really do not belong in this thread.

It's not really about you. Sorry if my pinging this issue up again somehow has hurt your feelings, but you must agree that it needs further attention.

Instead of communicating like this: Unfortunately, none of the entries on https://mxtoolbox.com/SuperTool.aspx?action=mx%3abackdropcms.org&run=toolpage has progressed.

I would suggest saying something more like: while I appreciate the improvements that have been made, can we continue to work on the goal of implementing dmarc or something like that.

[jenlampton]

@quicksketch you are the only one who can move this forward. Will you have time to update the DNS today?

[stpaultim]

@alanmels - FYI, this was discussed during dev meeting, both before we started recording and again during the recorded portion of the meeting. I was not particularly involved in this discussion, but you can watch the video of the meeting to see some of what was said. Hopefully, someone with more understanding of this issue will post a report.

[alanmels]

@larsdesigns, I am really sorry I was not polite enough. But please try to understand that all kinds of users could come in to the community, including those like myself who mastered English as not even second or third, but nth language. Some people feel just happy they can express themselves enough to get issues addressed.

When it comes to especially technical issues we'd like to be as brief and concise as possible. So please do not expect everyone popping up here is the most courteous gentleman in the world who will definitely express all kinds of gratitudes to you in advance before proceeding to the main matter. I don't believe saying "the issue unfortunately is still there" should be considered so impolite. But maybe you native speakers know better, but maybe also you are just taking this too close to your heart.

[quicksketch]

Thanks so much @larsdesigns for fixing up the reverse DNS configuration and generating new DKIM keys. I updated the DNS entries with the following:

Updated DKIM key: TXT 201911._domainkey v=DKIM1; h=sha256; k=rsa; s=email; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzORGOnrycJPDF5eSbIZoM1WqEL1gGaYbsSSykPZTD15pk5u8pG3SKvNLAwxWHbe0nIuWucO+fmNp7J2sGrVduxF68wjemrbbJytk0LZIcUckBc0c7Pn85d8Bb25NYLXmCLOVKpwxyj7khTWoqh0UlNYYqlEFgBnwVzdCfH5n3crZ/iC/UAaJwSDZLQg5zCQ4X8q6QD5HDxGbeY

New DMARC policy: TXT _dmarc v=DMARC1; p=quarantine; pct=100; rua=mailto:abuse@backdropcms.org

Adding to general email problems on backdropcms.org, it appears that we had been using Sparkpost SMTP at one point, but it is no longer working. My experience with Sparkpost is not positive. My guess is they disabled our account without notifying us, or that they did notify us but their own emails got caught by Namecheap's spam filters (ironically enough) which prevented us from receiving them.

When Sparkpost stopped working, that may have resulted in our DKIM keys not matching any more, since our server wasn't configured to send email, it assumed Sparkpost would send them.

So our current situation is that we now have DKIM, SPF, MX, and DMARC records all set up. However, this still won't guarantee full deliverability because we're sending from our Linode server, which some mail providers may decide to discard anyway.

Let's see how email deliverability goes for the next few weeks and if necessary we can set up SMTP again, preferably with a different provider than Sparkpost.

[oadaeh]

However, this still won't guarantee full deliverability because we're sending from our Linode server, which some mail providers may decide to discard anyway.

If the DKIM key, email headers, and DNS entries are set up correctly, the providers should not be discarding the emails due to validity. If they are, then they're doing it wrong. That doesn't mean they still won't decide the content is spammy.

[jenlampton]

some mail providers may decide to discard anyway.

This is likely, since the IP address of this server had been blacklisted (for sure in Russia and Ukraine, and probably also other countries) before we moved onto this new Linode server.

Can we set up another SMTP provider right away rather than waiting? Is there a downside?

[larsdesigns]

Let me know which smtp relay service is decided upon. I can help with the configuration.

[jenlampton]

@quicksketch says the live site is already set up with MailJet, all we need to do is turn SMTP on (not the module, the setting). We already have DKIM/SPF records for MailJet at the DNS level.

[jenlampton]

I've enabled it, and sent a test email. It landed safely in my inbox.

[alanmels]

Hello Everyone! The https://mxtoolbox.com/SuperTool.aspx?action=mx%3abackdropcms.org&run=toolpage finally showing all items in green, yay! I've just tested this by retrieving a new password and the mail has been delivered showing no issues. Thank you all, especially @larsdesigns for resolving this issue!