SSL on fast server

[pjv]

When I enable SSL for a tracking domain, a Let's Encrypt cert is successfully received (and I can see that the cert exists in the postal db).

Connecting to the server on that tracking domain via http works fine:

curl http://click.my.tracking.domain Hello.

Connecting via https doesn't work:

curl https://click.my.tracking.domain curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to click.my.tracking.domain:443

Oops. Apparently same as #788

I followed the steps in https://github.com/atech/postal/wiki/Click-&-Open-Tracking

[If it matters, this is a DO server and the fast server is listening on (bound to) an "anchor IP" address for a DO floating IP. All of this is happening in IPV4 space as I have disabled IPV6 on the server. I'm thinking that probably doesn't have anything to do with it since the plumbing apparently works fine for non-ssl connections.]

And...

sudo netstat -lnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 10.10.0.6:80            0.0.0.0:*               LISTEN      1565/[postal] fast.
tcp        0      0 X.X.X.X:80              0.0.0.0:*               LISTEN      1090/nginx: master
tcp        0      0 0.0.0.0:4369            0.0.0.0:*               LISTEN      1158/epmd
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      755/systemd-resolve
tcp        0      0 10.10.0.6:443           0.0.0.0:*               LISTEN      1565/[postal] fast.
tcp        0      0 X.X.X.X.1:443           0.0.0.0:*               LISTEN      1090/nginx: master
tcp        0      0 0.0.0.0:22111           0.0.0.0:*               LISTEN      1020/sshd
tcp        0      0 127.0.0.1:5000          0.0.0.0:*               LISTEN      1562/[postal] web.1
tcp        0      0 0.0.0.0:25672           0.0.0.0:*               LISTEN      954/beam.smp
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      1315/mysqld

[willpower232]

Interesting, it looks like everything is working apart from the actual SSL certificate part.

Which OS and ruby version are you using? Are you using the latest version of Postal?

[pjv]

I installed postal 3 or 4 days ago using the quick install script with slight modifications (due to running on ubuntu 18.04 with ruby 2.5). Is there an easy way to tell if the codebase I installed is the latest version?

[willpower232]

Ah I have been running an Ubuntu 18.04 with Ruby 2.5 as a testing installation for a little while and I also can see your error message if I turn on click tracking.

Previously, we haven't been able to get to this point because of #674.

Unfortunately there isn't official support for this setup (#777) but hopefully it is coming as @catphish mentioned in #755.

FWIW Postal seems to work fully apart from HTTPS click tracking so you should be able to continue without it or alternatively get an Ubuntu 16.04 server going or mess around with your servers Ruby versions in the meantime.

[pjv]

@willpower232 Thanks for the followup. I'm not wanting to run a new production service on EOL software so I'll turn SSL tracking off for now and hope that future updates will bring postal's dependencies more up to date.

[jaydrogers]

I have been running this on Ubuntu 18.04 without any issues.

Just a heads up, I have run into issues with "Floating IP Addresses" with Digital Ocean before (with software other than Postal). Digital Ocean uses some form of NAT (they use Openstack) for floating IPs which can break certain things on some software.

I use Vultr for mine and you can find more of that story here: https://github.com/521dimensions/ansible-postal#-important-note-regarding-digital-oceans-floating-ip-addresses

[willpower232]

Great work, thanks for sharing!

You're including ruby 2.3 from an alternate source so I'm guessing this problem lies somewhere in using ruby 2.5 which is the default on ubuntu 18.04.

[jaydrogers]

Thanks!

I will do some testing with the Ruby versions and see if I can isolate where the issue happens. I went off of these instructions, but I just checked and noticed Ruby 2.3 is EOL since March 2019 😱

I will keep you posted.

[willpower232]

Thanks for your efforts! Will definitely be good to take a stab at this if atech haven't already

[jaydrogers]

So my testing with the default ruby version on Ubuntu 18.04 failed. It fails on this command:

postal bundle /opt/postal/vendor/bundle

Looking at the rest of the issues, looks like this is an ongoing discussion and a known issue?

• 684

• 755

• 777

[pjv]

When I was running 2.5 on 18.04, the bundle error message that I got had a suggestion for fixing it before or after the traceback which I did. I can't recall now the exact command but it was something along the lines of having the wrong version of bundle in the app's deps and going into /opt/postal/ or maybe /opt/postal/app and doing some kind of generic "rebundling" of the app (you can tell just about how much ruby development I've done) which I think included downloading a more recent bundle version.

Anyway, doing whatever that was got me past that error in the install and I was able to then run postal on ruby 2.5. As far as I could tell, everything seemed to work fine EXCEPT for SSL on the tracking server.

(re-opening this issue to track).

[pjv]

@jaydrogers FWIW, I set up a new server on vultr, following your tutorial and it seemed great the way they let you simply assign an additional IP and even give it a PTR record. I thought I was going to be able to really do exactly what I wanted (which also included segregating SMTP into two different sending servers, each with their own IP address).

It was all looking great until I built a test mailserver in postal and found that it couldn't send any mail at all. I banged my head against what I could have done wrong for a few hours until I looked into vultr's help system and found that by default they block outgoing traffic on port 25. I asked them to unblock it for me and they declined.

[willpower232]

@jaydrogers can't reproduce your issue I'm afraid, did the install succeed completely up to that point?

@pjv I presume that means you can't run telnet outlook-com.olc.protection.outlook.com 25 on your vultr server?

[pjv]

@willpower232 exactly. Trying to telnet on port 25 from the command line to my own primary mailserver (on another host) is how I finally realized that there was no problem with the postal install and something else was going on.

[willpower232]

That is annoying but interesting that it wasn't a problem for jay. Fortunately, other providers are available!

[pjv]

Fortunately, other providers are available!

I use Linode and DO. Linode makes it really hard to get a secondary IP address. DO gives you floating IPs that they will not let you set a PTR record on. I had never used Vultr before, until @jaydrogers' rec, so I just created an account yesterday and proceeded to set up the new server. The recency of the account creation might have had something to do with their declining to lift the port 25 ban, though what seemed to put them over the edge was my telling them that part of the email traffic that would be passing through my server was an opt-in newsletter that goes out every couple weeks from one of my clients. I understand wanting to be very careful about sending unsolicited email, but my firm has been around for a while and we never allow sending unsolicited email from any of our servers. Ever.

What was very nice about Vultr was that they let you simply assign up to 2 additional v4 IPs to a server right from the control panel and you can quickly and easily assign a PTR record to each of them. All exactly as it should work with a couple clicks.

Anyone know of another host that gives you that kind of flexibility AND let's you send outbound traffic on port 25?

[willpower232]

@pjv our only experience has been doing this on AWS and its worked out okay so far

@jaydrogers my mistake, I had "fixed" the problems by manually installing the older bundler gem uninstall bundler && gem install bundler:1.17.2 and then I found I also needed apt install zlib1g-dev in order to compile nokogiri

[pjv]

our only experience has been doing this on AWS and its worked out okay so far

Thanks. I'm an Amazon boycotter due to their [Orwellian] rekognition product.

[jaydrogers]

Response to @pjv:

It was all looking great until I built a test mailserver in postal and found that it couldn't send any mail at all. I banged my head against what I could have done wrong for a few hours until I looked into vultr's help system and found that by default they block outgoing traffic on port 25.

You've got to be kidding me 🤦‍♂️ I spend all of that time documenting it for Vultr, and I never ran into that issue because I created our account years ago. I am sure since you were a new account, you are at a "heightened risk" and thats why the ports are blocked you. I think that is garbage on their part because they specifically say that you can get it unblocked https://www.vultr.com/docs/what-ports-are-blocked

Thanks for letting me know, I updated that in the notes: https://github.com/521dimensions/ansible-postal#recommended-providers

Linode makes it really hard to get a secondary IP address.

Their new panel doesn't make it look too bad now. Linode is great as well. I've had really good luck with them too.

DO gives you floating IPs that they will not let you set a PTR record on. Just as long as you are using your main IP address to send email, you should be fine. If you send through many IPs on the same server, maybe try Linode or push back on Vultr to unblock 25?

Response to @willpower232:

my mistake, I had "fixed" the problems by manually installing the older bundler gem uninstall bundler && gem install bundler:1.17.2 and then I found I also needed apt install zlib1g-dev in order to compile nokogiri

Thanks for letting me know. It will be interesting to see where Postal goes from here. Before I start hacking away to get things to install, I think I might wait to see if it gets updated by the core team. I think once they get a chance to take a look at everything, a modernized Postal release will go a long way.

Once we hit that, it would nice to get some official Docker support to make deployments even easier 😀

[pjv]

@jaydrogers

Thanks for letting me know, I updated that in the notes

Here's a few more notes for your notes 😄

Vultr

I had a lengthy exchange with Vultr tech support about unblocking outgoing port 25 traffic. They were very responsive. I'm fairly certain from that exchange that the real problem was that I told them that one of my clients would be sending out a bi-weekly newsletter to an opt-in list. They were adamant about not letting that kind of email go through, saying this:

Unfortunately, what you have described will not work out on our platform as we have a strict policy against anything even remotely resembling bulk email such as double/triple opt-in or otherwise.

Linode

I've been a Linode user for a long, long time. There is no technical issue with requesting and configuring additional IP addresses from them. It's a policy thing. I have done it before. First you have to open a support ticket to request the IP address. Then you have to justify your use of it and their guidelines for its use are pretty draconian. One of my reasons for wanting to set up postal on Vultr was because I want to segregate sending via two separate IP's (using postal's IP Pools feature). That's why I was excited to see that it was both easy to get a secondary IP on Vultr and also easy to PTR it. From my past interactions with Linode about secondary IP's, I already know that my reason for wanting it in this case would not be adequate for their policy and I don't want to lie to them about it.

DO

DO lets you send via port 25 on IPv4 from the main IP address and you can easily enough set up the floating IP for your fast / tracking server. With ruby2.3 that works fine via SSL as well (though I don't like running code on EOL software). Unfortuantely, you cannot PTR a floating IP so that makes sending via that IP impossible.

But the bigger problem with DO is that their ENTIRE block of IPv4 addresses are blocked by default by huge swaths of the email-receiving Internet. I spun up and tossed away a couple droplets in different NOCs until I was assigned a main and a floating IP that were both all green on two different RBL blacklist checkers. Didn't matter. MSN / Outlook / Hotmail block mail from the server because:

550 5.7.1 Unfortunately, messages from [X.X.X.X] weren't sent. Please contact your Internet service provider since part of their network is on our block list (S3140). You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors. [BN3NAM01FT039.eop-nam01.prod.protection.outlook.com]

sbcglobal / charter / ATT / etc. say this:

553 5.3.0 alph730 DNSBL:ATTRBL 521< X.X.X.X >_is_blocked.For assistance forward this email to abuse_rbl@abuse-att.net

I'm working on mitigating those server blocks now, but of course it's a PITA.

[willpower232]

@pjv good luck on your quest!

@jaydrogers FWIW I haven't found anything else wrong with it after installing the different bundler and the one missing package apart from the click tracking SSL but yeah I'm keeping my fingers crossed for an official version soon, particularly with the upcoming 20.04 LTS

[jaydrogers]

Man, what a train wreck. I am starting a conspiracy theory that Google and Microsoft make self hosting email difficult so that they can mine and sell your data easier 😀

Thanks for the insight @pjv. Currently our biggest production instance is in a datacenter of an ISP (for their own use), so we haven't had any major issues.

I assume you know about this, but I am posting incase someone else lands on the thread. These are "must have tools" for self hosting email:

• https://sendersupport.olc.protection.outlook.com/SNDS/index.aspx
• https://www.gmail.com/postmaster/

I will definitely take your experiences into consideration on deploying Postal anywhere else. Thanks for sharing!

[pjv]

Before I start hacking away to get things to install, I think I might wait to see if it gets updated by the core team. I think once they get a chance to take a look at everything, a modernized Postal release will go a long way.

Once we hit that, it would nice to get some official Docker support to make deployments even easier

I'm with you there. I tried to use this docker postal -> https://github.com/CatDeployed/docker-postal but it seemed useless. I'd love to see a proper docker install.

I've also been thinking about building a WordPress plugin to send mail via postal using the API, but I want to see whether the core team are still into developing / updating it before I spend a lot of time on a project like that.

[pjv]

Man, what a train wreck. I am starting a conspiracy theory that Google and Microsoft make self hosting email difficult so that they can mine and sell your data easier

Actually, as usual, Google is smarter about that than MS. I've had no trouble whatsoever delivering to gmail addresses. If you set up proper DNS records, google will accept and deliver your mail. That lets them spy on you all the more.

[kanadaj]

@willpower232 It seems to me that the original issue is still not resolved - I'm also having trouble getting SSL tracking working on Fast Server on 18.04.

fast_server:
  # This can be enabled to enable click & open tracking on emails. It is disabled by
  # default as it requires a separate static IP address on your server.
  enabled: true
  bind_address: 0.0.0.0
  port: 5010
  ssl_port: 5011

# curl --insecure -v https://127.0.0.1:5011
* Rebuilt URL to: https://127.0.0.1:5011/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 5011 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:5011
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:5011

# netstat -lnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      3870/nginx: master
tcp        0      0 0.0.0.0:5010            0.0.0.0:*               LISTEN      6375/[postal] fast.
tcp        0      0 0.0.0.0:5011            0.0.0.0:*               LISTEN      6375/[postal] fast.

# curl http://127.0.0.1:5010
Hello.

[kanadaj]

@jaydrogers

So my testing with the default ruby version on Ubuntu 18.04 failed. It fails on this command:

postal bundle /opt/postal/vendor/bundle

FYI the fix for this is

gem update --system

based on https://bundler.io/blog/2019/05/14/solutions-for-cant-find-gem-bundler-with-executable-bundle.html which did work for me. There is no need to use an older bundler.

[jaydrogers]

Thanks for the notes @kanadaj! There are talks of modernizing the Ruby version for postal, so hopefully that will help soon too.

Thanks for sharing.

[SonicGD]

Hello. I was also having this problem on newer versions of ruby. In fact, i've tried to build multiple docker images with different ruby base image versions:

• ruby:2.3-alpine - fast server ssl works
• ruby:2.4-alpine - fast server ssl doesn't works
• ruby:2.6-buster - fast server ssl doesn't works
• ruby:2.4-buster - fast server ssl doesn't works
• ruby:2.3 stretch - fast server ssl works

So, only ruby 2.3 will give you working fast server ssl. Really waiting for postal to upgrade ruby.

[willpower232]

@SonicGD I've just successfully created a tracking certificate using the latest code from the repo and the ruby:2.6 based Dockerfile.

You need to have registered the private key (config/lets_encrypt.pem) with lets encrypt using postal register-lets-encrypt youremail@example.com.

If the certificate fails to be renewed or generated, it will not be attempted again automatically for another 24 hours.

You can access the track_certificates database table to change the renew time if you wish and then either wait for the cron to execute the command to queue up the certificate or run the command yourself.

If the track_certificates table has a verification_path and a verification_string, that implies that you are definitely registered with Lets Encrypt so if you are still having problems, make sure that your click tracking domain is completely web accessible and that visiting the domain you are trying to make click trackable outputs Hello.

When the certificate fetching fails, the worker log should contain the phrase Status was not valid (was: X) where X may be "invalid" or another value to say that Lets Encrypt was unable to connect to the domain.

Hope this helps, if you're still having problems then open a new issue with some more details from the logs about what is happening.

[SonicGD]

@willpower232 My problem is not with certificate generation, it works without problem. I'm talking about problem described in issue first message. When i start fast server on ruby > 2.3 ssl port just doesn't work. I get the same curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL errors trying to connect to this port.

[willpower232]

My mistake, it seems that there is a deprecated callback in use here:

https://github.com/postalhq/postal/blob/a66b94b228204fa9f389191825b4e0e7fdff754a/lib/postal/fast_server/client.rb#L155-L157

Removing it appears to resolve the situation for newer ruby versions but presumably it was there for a reason so hopefully there shall be a real fix shortly.