search

haraka / Haraka

A fast, highly extensible, and event driven SMTP server
https://haraka.github.io
MIT License
5.09k stars 661 forks source link

smtp_forward plugin seems to not work #1318

Closed pjebs closed 8 years ago

pjebs commented 8 years ago

I am testing using smtp_forward using my SES credentials. These credentials work for all my other projects:

; host to connect to host=email-smtp.us-west-2.amazonaws.com ; ; port to connect to port=465 ; ; timeout backend connection from pool ;timeout=300 ; ; max connections in pool ;max_connections=1000 ; ; uncomment to enable TLS to the backend SMTP server enable_tls=1 ; ; for messages that have multiple RCPT, send a separate message for each RCPT ; when forwarding. ;one_message_per_rcpt=true ; ; uncomment to use smtp client authorization auth_type=plain auth_user=XXX auth_pass=XXX

pjebs commented 8 years ago

Haraka seems to be saying something is wrong with the plugin:

[DEBUG] [597C409B-0419-4D70-9AEF-555DE32F197E.1] [core] running queue hooks [DEBUG] [597C409B-0419-4D70-9AEF-555DE32F197E.1] [core] running queue hook in queue/smtp_forward plugin [INFO] [597C409B-0419-4D70-9AEF-555DE32F197E.1] [queue/smtp_forward] forwarding to email-smtp.us-west-2.amazonaws.com:465 [INFO] [-] [core] [smtp_client_pool] [465:email-smtp.us-west-2.amazonaws.com:300] dispense() clients=1 available=0 [DEBUG] [-] [core] [smtp_client_pool] [465:email-smtp.us-west-2.amazonaws.com:300] createResource() - creating obj - count=1 min=0 max=1000 [DEBUG] [-] [core] [smtp_client_pool] uuid=168C66FC-4901-40FF-93BD-B168474A21DD host=email-smtp.us-west-2.amazonaws.com port=465 pool_timeout=300 created [DEBUG] [597C409B-0419-4D70-9AEF-555DE32F197E.1] [queue/smtp_forward] Got smtp_client: 168C66FC-4901-40FF-93BD-B168474A21DD [CRIT] [597C409B-0419-4D70-9AEF-555DE32F197E.1] [core] Plugin queue/smtp_forward failed: TypeError: Cannot read property 'host' of undefined at smc_cb (/usr/local/lib/node_modules/Haraka/plugins/queue/smtp_forward.js:55:96) at /usr/local/lib/node_modules/Haraka/smtp_client.js:434:9 at callback_wrapper (/usr/local/lib/node_modules/Haraka/smtp_client.js:277:17) at /usr/local/lib/node_modules/Haraka/node_modules/generic-pool/lib/generic-pool.js:291:11 at Object.generic_pool.Pool.create (/usr/local/lib/node_modules/Haraka/smtp_client.js:252:17) at createResource (/usr/local/lib/node_modules/Haraka/node_modules/generic-pool/lib/generic-pool.js:269:13) at dispense (/usr/local/lib/node_modules/Haraka/node_modules/generic-pool/lib/generic-pool.js:261:9) at Object.me.acquire (/usr/local/lib/node_modules/Haraka/node_modules/generic-pool/lib/generic-pool.js:329:5) at Object.pool.acquire (/usr/local/lib/node_modules/Haraka/smtp_client.js:279:21) at Object.exports.get_client_plugin (/usr/local/lib/node_modules/Haraka/smtp_client.js:308:10) [DEBUG] [597C409B-0419-4D70-9AEF-555DE32F197E.1] [core] hook=queue plugin=queue/smtp_forward function=hook_queue params="" retval=CONT msg="" [NOTICE] [597C409B-0419-4D70-9AEF-555DE32F197E.1] [core] queue code=CONT msg=" (597C409B-0419-4D70-9AEF-555DE32F197E.1)" [PROTOCOL] [597C409B-0419-4D70-9AEF-555DE32F197E.1] [core] S: 451 (597C409B-0419-4D70-9AEF-555DE32F197E.1) [DEBUG] [597C409B-0419-4D70-9AEF-555DE32F197E.1] [core] running reset_transaction hooks [PROTOCOL] [597C409B-0419-4D70-9AEF-555DE32F197E.1] [core] C: QUIT state=1 [DEBUG] [597C409B-0419-4D70-9AEF-555DE32F197E.1] [core] running quit hooks [PROTOCOL] [597C409B-0419-4D70-9AEF-555DE32F197E.1] [core] S: 221 Roman-Chronos closing connection. Have a jolly good day. [DEBUG] [597C409B-0419-4D70-9AEF-555DE32F197E.1] [core] running disconnect hooks [NOTICE] [597C409B-0419-4D70-9AEF-555DE32F197E.1] [core] disconnect ip=192.168.99.1 rdns="DNSERROR" helo="192.168.99.100" relay=N early=N esmtp=Y tls=N pipe=N errors=0 txns=1 rcpts=1/0/0 msgs=0/1/0 bytes=219 lr="451 (597C409B-0419-4D70-9AEF-555DE32F197E.1)" time=4.368

msimerson commented 8 years ago

See if this patch helps:

diff --git a/plugins/queue/smtp_forward.js b/plugins/queue/smtp_forward.js
index a8884fb..f137fea 100644
--- a/plugins/queue/smtp_forward.js
+++ b/plugins/queue/smtp_forward.js
@@ -63,7 +63,7 @@ exports.hook_queue = function (next, connection) {
         smtp_client.next = next;

         if (cfg.auth_user) {
-            connection.loginfo(plugin, 'Configuring authentication for SMTP server ' + cfg.main.host + ':' + cfg.main.port);
+            connection.loginfo(plugin, 'Configuring authentication for SMTP server ' + cfg.host + ':' + cfg.port);
             smtp_client.on('greeting', function() {

                 var base64 = function(str) {
pjebs commented 8 years ago

I'm not able to test this code unfortunately. I hired a freelancer to developer the code and he was meant to do something like: https://gist.github.com/smfreegard/ff79d02aeb94b9065359

I just tried to set up smtp_forward.ini just to test out his system, but he will now fix up the queue_outbound, so I wouldn't know how to implement your changes above.

baudehlo commented 8 years ago

The beauty of Haraka is you can easily patch this in. Just copy the core plugin to your install plugins/ folder and edit it there. It will be loaded in preference to the core one.

On Jan 23, 2016, at 9:01 PM, pjebs notifications@github.com wrote:

I'm not able to test this code unfortunately. I hired a freelancer to developer the code and he was meant to do something like: https://gist.github.com/smfreegard/ff79d02aeb94b9065359

I just tried to set up smtp_forward.ini just to test out his system, but he will now fix up the queue_outbound, so I wouldn't know how to implement your changes above.

— Reply to this email directly or view it on GitHub.

pjebs commented 8 years ago

I tried the patch. I don't see the plugin failed error but it still doesn't seem to connect to aws ses.

pjebs commented 8 years ago

[queue/smtp_forward] forwarding to email-smtp.us-west-2.amazonaws.com:465 [INFO] [-] [core] [smtp_client_pool] [465:email-smtp.us-west-2.amazonaws.com:300] dispense() clients=1 available=0 [DEBUG] [-] [core] [smtp_client_pool] [465:email-smtp.us-west-2.amazonaws.com:300] createResource() - creating obj - count=1 min=0 max=1000 [DEBUG] [-] [core] [smtp_client_pool] uuid=AD8D6F85-6842-4C8D-B991-22CB2F669300 host=email-smtp.us-west-2.amazonaws.com port=465 pool_timeout=300 created [DEBUG] [C01AD2C3-A5CA-4F89-B7B4-08EF7DC8189C.1] [queue/smtp_forward] Got smtp_client: AD8D6F85-6842-4C8D-B991-22CB2F669300 [INFO] [C01AD2C3-A5CA-4F89-B7B4-08EF7DC8189C.1] [queue/smtp_forward] Configuring authentication for SMTP server email-smtp.us-west-2.amazonaws.com:465 [CRIT] [C01AD2C3-A5CA-4F89-B7B4-08EF7DC8189C.1] [core] Plugin queue/smtp_forward timed out on hook queue - make sure it calls the callback [INFO] [C01AD2C3-A5CA-4F89-B7B4-08EF7DC8189C.1] [core] hook=queue plugin=queue/smtp_forward function=hook_queue params="" retval=DENYSOFT msg="plugin timeout" [DEBUG] [C01AD2C3-A5CA-4F89-B7B4-08EF7DC8189C.1] [core] running deny hooks [NOTICE] [C01AD2C3-A5CA-4F89-B7B4-08EF7DC8189C.1] [core] queue code=DENYSOFT msg="plugin timeout (C01AD2C3-A5CA-4F89-B7B4-08EF7DC8189C.1)" [PROTOCOL] [C01AD2C3-A5CA-4F89-B7B4-08EF7DC8189C.1] [core] S: 450 plugin timeout (C01AD2C3-A5CA-4F89-B7B4-08EF7DC8189C.1) [DEBUG] [C01AD2C3-A5CA-4F89-B7B4-08EF7DC8189C.1] [core] running reset_transaction hooks [PROTOCOL] [C01AD2C3-A5CA-4F89-B7B4-08EF7DC8189C.1] [core] C: QUIT state=1 [DEBUG] [C01AD2C3-A5CA-4F89-B7B4-08EF7DC8189C.1] [core] running quit hooks [PROTOCOL] [C01AD2C3-A5CA-4F89-B7B4-08EF7DC8189C.1] [core] S: 221 Roman-Chronos closing connection. Have a jolly good day. [DEBUG] [C01AD2C3-A5CA-4F89-B7B4-08EF7DC8189C.1] [core] running disconnect hooks [NOTICE] [C01AD2C3-A5CA-4F89-B7B4-08EF7DC8189C.1] [core] disconnect ip=192.168.99.1 rdns="DNSERROR" helo="pjs-imac.local" relay=N early=N esmtp=Y tls=N pipe=N errors=0 txns=1 rcpts=1/0/0 msgs=0/1/0 bytes=219 lr="450 plugin timeout (C01AD2C3-A5CA-4F89-B7B4-08EF7DC8189C.1)" time=34.428 [DEBUG] [-] [core] [smtp_client_pool] AD8D6F85-6842-4C8D-B991-22CB2F669300 destroyed, state=2 [DEBUG] [-] [core] [smtp_client_pool] AD8D6F85-6842-4C8D-B991-22CB2F669300: SMTP connection ended (state=4) [DEBUG] [-] [core] [smtp_client_pool] AD8D6F85-6842-4C8D-B991-22CB2F669300: SMTP connection closed (state=4)

smfreegard commented 8 years ago

Use port 25 or 587. Don't use 465 (SMTPS) - for one it's deprecated (port 587 should be used instead) and secondly it doesn't look like it's supported anywhere other than on an inbound listener socket in Haraka, so smtp_proxy/smtp_forward/outbound etc. won't work with it.

pjebs commented 8 years ago

Okay I'll change to port 587, but I used those credentials (including 465) everywhere in other projects and it work.

Could you explain the "it doesn't look like it's supported anywhere other than on an inbound listener socket in Haraka, so smtp_proxy/smtp_forward/outbound etc. won't work with it."

smfreegard commented 8 years ago

Like I said SMTPS (465/tcp) is deprecated in favour of using SMTP Submission (587/tcp) and therefore it wasn't added in most places in Haraka except for the special case of being able to set 'listen=[::0]:465 in smtp.ini for Haraka to listen on the port so it could support legacy clients.

Specifically - I've just looked at smtp_client.js which is used by smtp_proxy and smtp_forward and there's no special provision for SMTPS in that code, nor is there any in outbound.js.

In reality there's no issue being as everywhere that supports 465/tcp does so for legacy reasons (crap old Microsoft Clients) also supports 587/tcp, so you just use that instead which is largely why there's no handling of it in smtp_client/outbound in Haraka.

If you're wondering why it requires special provision - then it's simply 465/tcp is like HTTPS - it requires TLS negotiation at connection whereas 25/tcp and 587/tcp do not - they use STARTTLS instead.

pjebs commented 8 years ago

I made the required settings and swaks complains about STARTTLS which you mentioned at the very end.

-> Subject: test Mon, 25 Jan 2016 21:59:14 +1100 -> X-Mailer: swaks v20130209.0 jetmore.org/john/code/swaks/ -> -> This is a test mailing -> -> . <\ 550 Must issue a STARTTLS command first (8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1) -> QUIT <- 221 Roman-Chronos closing connection. Have a jolly good day.

pjebs commented 8 years ago

is this a bug with Haraka's smtp plugin?

pjebs commented 8 years ago

[DEBUG] [8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1] [core] running queue hook in queue/smtp_forward plugin [INFO] [8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1] [queue/smtp_forward] forwarding to email-smtp.us-west-2.amazonaws.com:587 [INFO] [-] [core] [smtp_client_pool] [587:email-smtp.us-west-2.amazonaws.com:300] dispense() clients=1 available=0 [DEBUG] [-] [core] [smtp_client_pool] [587:email-smtp.us-west-2.amazonaws.com:300] createResource() - creating obj - count=1 min=0 max=1000 [DEBUG] [-] [core] [smtp_client_pool] uuid=305668E9-A6C8-42C5-867B-E7C1745D1E6D host=email-smtp.us-west-2.amazonaws.com port=587 pool_timeout=300 created [DEBUG] [8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1] [queue/smtp_forward] Got smtp_client: 305668E9-A6C8-42C5-867B-E7C1745D1E6D [INFO] [8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1] [queue/smtp_forward] Configuring authentication for SMTP server email-smtp.us-west-2.amazonaws.com:587 [PROTOCOL] [8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1] [queue/smtp_forward] S: 220 email-smtp.amazonaws.com ESMTP SimpleEmailService-1207632523 cy0IX3GgdI7hppRr10mm\r\n [PROTOCOL] [8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1] [queue/smtp_forward] C: EHLO Roman-Chronos [INFO] [8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1] [queue/smtp_forward] Authenticating with AUTH PLAIN XXX [PROTOCOL] [8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1] [queue/smtp_forward] C: AUTH PLAIN XXX [PROTOCOL] [8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1] [queue/smtp_forward] S: 250-email-smtp.amazonaws.com\r\n [PROTOCOL] [8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1] [queue/smtp_forward] S: 250-8BITMIME\r\n [PROTOCOL] [8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1] [queue/smtp_forward] S: 250-SIZE 10485760\r\n [PROTOCOL] [8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1] [queue/smtp_forward] S: 250-STARTTLS\r\n [PROTOCOL] [8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1] [queue/smtp_forward] S: 250-AUTH PLAIN LOGIN\r\n [PROTOCOL] [8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1] [queue/smtp_forward] S: 250 Ok\r\n [PROTOCOL] [8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1] [queue/smtp_forward] C: MAIL FROM:pj@sky.com [PROTOCOL] [8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1] [queue/smtp_forward] S: 530 Must issue a STARTTLS command first\r\n [INFO] [8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1] [core] hook=queue plugin=queue/smtp_forward function=hook_queue params="" retval=DENY msg="Must issue a STARTTLS command first" [DEBUG] [8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1] [core] running deny hooks [NOTICE] [8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1] [core] queue code=DENY msg="Must issue a STARTTLS command first (8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1)" [PROTOCOL] [8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1] [core] S: 550 Must issue a STARTTLS command first (8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1) [DEBUG] [8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1] [core] running reset_transaction hooks [DEBUG] [-] [core] [smtp_client_pool] 305668E9-A6C8-42C5-867B-E7C1745D1E6D resetting, state=2 [PROTOCOL] [8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1] [core] C: QUIT state=1 [DEBUG] [8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1] [core] running quit hooks [PROTOCOL] [8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1] [core] S: 221 Roman-Chronos closing connection. Have a jolly good day. [DEBUG] [8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1] [core] running disconnect hooks [NOTICE] [8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1] [core] disconnect ip=192.168.99.1 rdns="DNSERROR" helo="pjs-imac.local" relay=N early=N esmtp=Y tls=N pipe=N errors=0 txns=1 rcpts=1/0/0 msgs=0/0/1 bytes=219 lr="550 Must issue a STARTTLS command first (8DBEC4C9-E465-469D-AE71-6414D9E17BB9.1)" time=6.736 [DEBUG] [-] [core] [smtp_client_pool] 305668E9-A6C8-42C5-867B-E7C1745D1E6D destroyed, state=3 [DEBUG] [-] [core] [smtp_client_pool] 305668E9-A6C8-42C5-867B-E7C1745D1E6D: SMTP connection closed (state=4)

smfreegard commented 8 years ago

is this a bug with Haraka's smtp plugin?

No - you need to generate a certificate pair (tls_key.pem and tls_cert.pem) and Haraka will negotiate STARTTLS automatically. See the docs for plugins/tls for details.

pjebs commented 8 years ago

I didn't know how to create the certificate pair properly so I just decided to use: https://gist.github.com/smfreegard/ff79d02aeb94b9065359

I'm trying to get haraka to send to my hotmail address.

Swaks seems to not complain: (i.e. 250 Message Queued (E5D6A75F-0629-4F2E-A643-A986B27168FF.1)

When I look in my Haraka log it it seems to say:

[ERROR] [E5D6A75F-0629-4F2E-A643-A986B27168FF.1.1] [outbound] Ongoing connection failed to 65.55.33.135:25 : Error: connect ECONNREFUSED 65.55.33.135:25 [INFO] [E5D6A75F-0629-4F2E-A643-A986B27168FF.1.1] [outbound] Attempting to deliver to: 207.46.8.199:25 (0) (1) [ERROR] [E5D6A75F-0629-4F2E-A643-A986B27168FF.1.1] [outbound] Ongoing connection failed to 65.55.33.135:25 : Error: connect ECONNREFUSED 65.55.33.135:25 [INFO] [E5D6A75F-0629-4F2E-A643-A986B27168FF.1.1] [outbound] Temp failing 1454199975656_3_23_1492.9e71fd6b53c5 for 512 seconds: Tried all MXs

It looks like it is in the queue and it continually tries and fails to send to hotmail. I tried with gmail too.

Is it unusual for hotmail and gmail to refuse a Haraka connection? Is it something to do with creating TLS certificates (even though I am now using outbound queue instead of smtp_forward?

smfreegard commented 8 years ago

The answer is simpler than that - port 25 appears to be firewalled.

I'm guessing you're using Amazon EC2 - if so, port 25 is severely throttled by default to prevent Spam from eminating from EC2. See https://aws-portal.amazon.com/gp/aws/html-forms-controller/contactus/ec2-email-limit-rdns-request

You don't need a TLS certificate when sending mail like this - but without one all mail will be sent unencrypted. It really is simple to generate a self-signed certificate and it's all documented here: https://github.com/haraka/Haraka/blob/master/docs/plugins/tls.md

pjebs commented 8 years ago

I was actually sending from my Mac using a local docker container running haraka. How can I send to hotmail or gmail then?

smfreegard commented 8 years ago

My answer remains the same - port 25 is being blocked somewhere either by your OS, router or maybe even your ISP.

You will have to do some troubleshooting youself to work out which.

pjebs commented 8 years ago

You were right about ISP blocking port 25. How can I disable Haraka from sending this email:

Hi. This is the Haraka Mailer program at Email Forwarder. I'm afraid I wasn't able to deliver your message to the following addresses. This is a permanent error; I've given up. Sorry it didn't work out.

Intended Recipients: p@hotmail.com Failure Reason: Error: 550 DY-001 (BLU004-MC1F24) Unfortunately, messages from 122.333.69.80 weren't sent. Please contact your Internet service provider. You can tell them that Hotmail does not relay dynamically-assigned IP ranges. You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors.

Final recipient: rfc822;p@hotmail.com Action: failed Remote-MTA: mx4.hotmail.com Diagnostic-Code: smtp;550 DY-001 (BLU004-MC1F24) Unfortunately, messages from 122.333.69.80 weren't sent. Please contact your Internet service provider. You can tell them that Hotmail does not relay dynamically-assigned IP ranges. You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors.

smfreegard commented 8 years ago

That is a DSN message - it's required if you want to be compatible with the e-mail RFCs. Without these messages; e-mail messages can't be traced and it will seem to your users that you're losing their mail. They can be blocked, but you'll cause yourself a lot of issues if you do that without understanding the ramifications of doing so. Go and read the docs on hook_bounce (in docs/Outbound.md) and they should tell you how.

And if you look at the Failure Reason - it's because you're on a dynamic IP range from your ISP and Hotmail doesn't accept messages from dynamic IPs. e.g. your IP probably doesn't have a PTR record or it looks 'dynamic' based on the naming.

I wouldn't have accepted that message here either for the same reason.

pjebs commented 8 years ago

Thanks so much for your help. I didn't know sending an email was so complex.

I'm making an email forwarder:

Since emails could be coming in from anyone and they could have any content what is the best way to make sure the email correctly forwards - particularly for services like hotmail and gmail.

I hired a freelancer to create my project. The plugins he made are (mysql and mongodb):

I am willing to upload these plugins once finished to the Haraka project. Will they be accepted??? What is the process?

smfreegard commented 8 years ago

I didn't know sending an email was so complex.

E-mail is very complex thanks to the spam issue. There are a lot of potential obstacles to overcome.

I'm making an email forwarder:

Since emails could be coming in from anyone and they could have any content what is the best way to make sure the email correctly forwards - particularly for services like hotmail and gmail.

Like I think I mentioned before - you MUST use SRS (sender rewriting scheme) for forwarding. Your service will fail miserably otherwise.

When forwarding like you describe - the envelope From will remain as the original senders address - if this is not rewritten to an SRS address and the destination servers check SPF on the senders domain, then it will potentially fail or softfail and the forwarded message will be treated as spam, which will tarnish the reputation of your IP address which in turn will put other messages into spam folders for other domains.

I'd love to see an SRS plugin for Haraka, I've been meaning to write one but haven't had time to pursue it.

As for the other plugins - I'm sure they'll be useful, but as you didn't write them yourself they are probably of limited value inside the Haraka project as we would have to maintain and support them for others to use, so you're better off putting them in their own Github repo along with the docs as to how to use them.

pjebs commented 8 years ago

Can I avoid all of this if I use SES to forward or will using SES just complicate things further compared to just getting Haraka to send it off?

smfreegard commented 8 years ago

I don't know - I've never used Amazon SES. But I highly doubt it would be suitable for a forwarding service. I've only been on the receiving end of SES and as a receiver I'm less than impressed.

SRS was specifically designed for forwarding, so it is what I would use if I were building a forwarder....

BradRuderman commented 8 years ago

@smfreegard @pjebs Quick question, were you able to figure this out for 587. I am setting up smtp_forward from EC2 to SES, on port 587, I am receiving the STARTTLS error despite trying a bunch of recommendations on this thread.

pjebs commented 8 years ago

I actually put the project on hold so I never looked into it. It just got too complicated for me.