search

garageScript / curriculum

GarageScript Curriculum
21 stars 164 forks source link

[Tutorial] - How to setup a server that sends email #353

Open songz opened 3 years ago

songz commented 3 years ago

Intro

While trying to figure out how to replace c0d3.com's email system, I thought it would be nice to setup our own email servers for our students to use. This quickly threw me down a rabbit hole of how email servers work.

Setting up an email server is hard. When I started on this journey I didn't know what the keywords are to search for, making it difficult to find the content you need to understand how it works. If you simply looked up how to send emails, you will get pagefuls of tutorials telling you how to connect to gmail smtp, SES smtp, etc. Hard to find articles on how to setup your own server that sends emails, to run your own email delivery service.

This article is a documentation of the steps I took to run my own server that delivers my emails directly so I have a reference in case I need to set this up again in the future. Hopefully it helps someone out there.

This article should cover what is spf, dkim, and dmarc conceptually and how each of these are used as security measures to prevent spam.

This article also documents a few different tools that helped my debug and helped me make sure that my email server is configured correctly.

Steps

Below will be a list of steps that I took to set it up my own email server.

port 25?

Make sure Port 25 is turned on. Sending emails requires back and forth communication between servers on port 25. Because of spam, most hosting companies block port 25. Hostwinds (referral) is an example of a hosting company that allows port 25 by default (I tested this tutorial on their $5 server - Ubuntu 18.04).

After I ssh'd into server, I made sure to enable firewall on port 25: ufw allow 25

Setup and try

I used an application called sendmail to send email.

sudo apt install sendmail

With sendmail installed, I could have sent my first email! Tip: Do not send to a gmail or an email powered by google at this point. If you do this without your server configured correctly (covered in the later sections), your server's ip will be blacklisted and may take 2 weeks to remove after filling out this blacklist removal form.

Sample output of my ip adress being blocked by gmail. Notice the message Our system has detected an unusual rate of unsolicited mail originating from your IP address...

image

To make sure my sendmail application can deliver emails I used ethereal.email to generate emails on the fly and sent the account an email.

sendmail -v justice.jaskolski@ethereal.email
Subject: sample email subject here
email body goes here. 
.

Make sure you exit out of the program by putting . on a new line an hitting enter. You should see messages output from the console. If you do not, you can find the logs in /var/log/mail.log.

I was able to verify that the email arrived in the email inbox and here's a sample output from my console:

image

Becoming Legit

Now that I can send a shitty email (that will probably get my server immediately blacklisted), it is time to make it legit.

Domain & Server

First I need a domain name. I got my domain name from godaddy. For the rest of the example I will be using my domain name m8l.me. I want my email server to be recognizable by domain name (smtp.m8l.me) so I added an A record for smtp.m8l.me to point to the ip address of our server.

(optional) - You should set your mx record to your subdomain for sending emails: smtp.m8l.me. When other servers (like gmail) need to deliver emails to your domain, they will look at this record to figure out where to send the email to. This tutorial does not show you how to receive emails, but setting up this record helps legitimize your domain.

Record Host points to
A smtp 104.958.243.486
MX smtp smtp.m8l.me

Now when my server sends an email to gmail, gmail is going to do a reverse dns lookup to get domain name from the ip address of my server (Opposite from the usual dns lookup where it gets ip address from the domain name). To set this, I went to my hosting company where I got my server and configured the reverse DNS. Here is a direct link I used to configure my hostwind server reverse DNS record. I made sure both ipv4 and ipv6 ip addresses are configured with smtp.m8l.me.

To verify that reverse DNS is setup correctly, I ran dig -x 104.958.243.486 to see what domain name my ip address resolved to.

Now that my server is all mapped correctly to the domain, I need to make sure sendmail on my server identifies itself correctly when communicating with other email servers. Type hostname into the terminal on your server to see how it currently identifies itself. I want it to say smtp.m8l.me.

Here's what I did to set the hostname property on my server:

  1. Set hostnamectl: sudo hostnamectl set-hostname smtp.m8l.me
  2. Edit hostname file to reflect the new name: /etc/hostname
  3. Restart: hostnamectl

Lastly, I deleted all the records and added smtp.m8l.me to this file: /etc/mail/local-host-names.

spf

When my server sends an email to gmail servers, they will first do a reverse dns lookup from my server's ip address to get the domain name. With this domain name gmail servers will look up my domain name's spf info to verify that my server has the permission to send email on behalf of that domain name.

To lookup a site's spf info (like gmail servers) you can look it up by fetching all the TXT records for the domain. To do this, run nslookup -q=TXT smtp.m8l.me in the terminal to get the TXT records for smtp.m8l.me.

To set my spf info, I needed to set my domain name's TXT record:

host TXT Value
smtp v=spf1 ip4:104.958.243.486 include:smtp.m8l.me -all

In the above record:

To verify that your TXT records are properly set, run nslookup -q=TXT smtp.m8l.me and look for the record that starts with v=spf1....

dkim

After verifying that my server has the permission to deliver email, gmail servers now needs to get my security information. It does this by looking at the dkim info for my domain name, which should specify the encryption that the email will use and the public key. In order to set this, I first need to generate a key.

To generate a public / private key, I needed to install opendkim and then mint the key (2 new files).

sudo apt-get install opendkim opendkim-tools
opendkim-genkey -b 1024 -s mail -d smtp.m8l.me

I will get back 2 new files.

mail.txt specifies what how to set the dkim info for my domain name. Below is a sample output.

mail._domainkey    IN    TXT    ("v1-DKIM1; h=sha256; k=rsa; p=M...DAQAB") ; -- DKIM key default for smtp.m8l.me

mail.private is a private key that sendmail will use to encrypt outgoing emails.

DNS - public key

Based on the content in mail.txt, I filled out my TXT record on my domain name provider like this:

host TXT Value
mail._domainkey.smtp v=DKIM1; h=sha256; k=rsa; p=MIG...DAQAB

Remove mail.txt because we don't need it anymore: rm mail.txt

Encryption - private key

After gmail servers gets my public key from your dkim info of your domain name, they will now expect encrypted email from me. sendmail needs opendkim server to do all the encryption so I need to setup opendkim server and configure sendmail to encrypt emails with opendkim.

First I need to make sure I move my private key (mail.private) to the correct place. 

sudo mkdir /etc/opendkim
sudo mv mail.private /etc/opendkim/

Now I want to configure opendkim service to encrypt with the correct key and specify the correct domain.

vim /etc/opendkim.conf

Domain   smtp.m8l.me
KeyFile  /etc/opendkim/mail.private
Selector   mail

mail is the Selector I used to generate the original key.

In the same file, I need to specify the socket to run on the correct port. 

Socket   inet:8891@localhost

Make sure all other instances of Socket are commented out.

Now edit another opendkim config file: vim /etc/default/opendkim

SOCKET="inet:8891@localhost" # listen on loopback on port 8891

Start/Restart opendkim service: sudo systemctl restart opendkim or sudo systemctl start opendkim.

To make sure my server is running on the correct specified port (8891), I ran lsof -i:8891 to check.

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
opendkim 1920 opendkim 3u IPv4 81346 0t0 TCP localhost:8891 (LISTEN)

Lastly, I needed to configure sendmail to use this opendkim service to encrypt emails.

Sendmail

Append the following into /etc/mail/sendmail.mc

INPUT_MAIL_FILTER(`opendkim', `S=inet:8891@localhost')

Regenerate sendmail config file as root:

sudo -i
m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf

Restart sendmail

sudo systemctl restart sendmail

DMARC

Now I'm almost legit, there is one last step. Email servers like gmail will eventually get alot of requests that tries to pretend to be my domain. I need a DMARC info for my domain name so that gmail knows what to do when they receive these hostile emails that tries to impersonate my domain name.

To do this, I add a TXT record in my domain name's DNS records.

host txt value
_dmarc.smtp v=DMARC1;p=reject;pct=100;rua=alessandro.reichert@ethereal.email

Now that I have the key components of an email configured, it is time to test it.

Debugging

To make sure opendkim is configured correctly, I tailed dkim logs:

sudo tail -f /var/log/syslog | grep -i dkim

To check if I had specified everything correctly, I searched for email deliverability test (this is a key word) that helps me verify my records. The easiest and did not require any signups is www.mail-tester.com. I sent an email to the provided address and got a 8.9/10 on my report!

Clicking on each section would tell you how what you configured incorrectly and why.

image

You con now use nodemailer and configure it to use sendmail as the transport layer. Sample code:

const transporter = nodemailer.createTransport({
        sendmail: true,
        newline: "unix",
        path: "/usr/sbin/sendmail",
});
transporter.sendMail(
        {
                from: "sender@smtp.m8l.me",
                to: "test-m6bef2c36@srv1.mail-tester.com",
                subject: "Message with dmarc",
                text: "I hope this message gets delivered!",
        },
        (err, info) => {
                console.log('info envelope', info.envelope);
                console.log('message id', info.messageId);
                console.log('error', err)
        }
);

If you send with nodemailer, you should have a score of 10/10! The only thing that you can improve on at this point is to add a List-Unsubscribe header but that is only necessary when you send out regular newsletters.

Screen Shot 2020-09-09 at 9 26 32 AM

Conclusion

All the work I did above is so that I can send emails to common email providers like gmail, outlook, yahoo, etc. without being blocked. Summary of the steps I took:

DNS

Server Provider

Server

Troubleshoot

arvindrajnaidu commented 3 years ago

Brilliant write-up @songz

joshbeam commented 3 years ago

awesome @songz, I learned a lot