STARTTLS queue/smtp_forward though maybe smtp_client issue Not Working

[BradRuderman]

Haraka version

Haraka.js — Version: 2.8.0-alpha.7

Expected behavior

smtp_client will issue a "STARTTLS" command when talking on port 587, and therefore work (i think) with amazon SES

Observed behavior

I noticed that the smtp_forward plugin doesn't work with SES. I get the following error:

[NOTICE] [A380A9AE-4A27-49D8-944A-6A54C94C9D05.1] [core] message mid="<abcd@mess.com>" size=1458 rcpts=1/0/0 delay=0.002 code=CONT msg=""
[DEBUG] [A380A9AE-4A27-49D8-944A-6A54C94C9D05.1] [core] running queue_outbound hooks
[DEBUG] [A380A9AE-4A27-49D8-944A-6A54C94C9D05.1] [core] running queue_outbound hook in queue/smtp_forward plugin
[DEBUG] [A380A9AE-4A27-49D8-944A-6A54C94C9D05.1] [queue/smtp_forward]
[INFO] [A380A9AE-4A27-49D8-944A-6A54C94C9D05.1] [queue/smtp_forward] forwarding to email-smtp.us-west-2.amazonaws.com:587
[INFO] [-] [core] [smtp_client_pool] [587:email-smtp.us-west-2.amazonaws.com:300] dispense() clients=1 available=0
[DEBUG] [-] [core] [smtp_client_pool] [587:email-smtp.us-west-2.amazonaws.com:300] createResource() - creating obj - count=1 min=0 max=10
[DEBUG] [-] [core] [smtp_client_pool] uuid=4545934E-47CC-46AD-91A3-A5B79FCC3AD4 host=email-smtp.us-west-2.amazonaws.com port=587 pool_timeout=300 created
[DEBUG] [A380A9AE-4A27-49D8-944A-6A54C94C9D05.1] [queue/smtp_forward] Got smtp_client: 4545934E-47CC-46AD-91A3-A5B79FCC3AD4
[INFO] [A380A9AE-4A27-49D8-944A-6A54C94C9D05.1] [queue/smtp_forward] Configuring authentication for SMTP server email-smtp.us-west-2.amazonaws.com:587
[ERROR] [-] [core] client.socket.remoteAddress undefined!
[PROTOCOL] [A380A9AE-4A27-49D8-944A-6A54C94C9D05.1] [queue/smtp_forward] S: 220 email-smtp.amazonaws.com ESMTP SimpleEmailService-1383310303 FpbSwsjPMyHTnpexjboB\r\n
[PROTOCOL] [A380A9AE-4A27-49D8-944A-6A54C94C9D05.1] [queue/smtp_forward] C: EHLO ip-172-31-16-162
[INFO] [A380A9AE-4A27-49D8-944A-6A54C94C9D05.1] [queue/smtp_forward] Authenticating with AUTH PLAIN AKIAMSECRET
[PROTOCOL] [A380A9AE-4A27-49D8-944A-6A54C94C9D05.1] [queue/smtp_forward] C: AUTH PLAIN SECRETPASSWORD
[PROTOCOL] [A380A9AE-4A27-49D8-944A-6A54C94C9D05.1] [queue/smtp_forward] S: 250-email-smtp.amazonaws.com\r\n
[PROTOCOL] [A380A9AE-4A27-49D8-944A-6A54C94C9D05.1] [queue/smtp_forward] S: 250-8BITMIME\r\n
[PROTOCOL] [A380A9AE-4A27-49D8-944A-6A54C94C9D05.1] [queue/smtp_forward] S: 250-SIZE 10485760\r\n
[PROTOCOL] [A380A9AE-4A27-49D8-944A-6A54C94C9D05.1] [queue/smtp_forward] S: 250-STARTTLS\r\n
[PROTOCOL] [A380A9AE-4A27-49D8-944A-6A54C94C9D05.1] [queue/smtp_forward] S: 250-AUTH PLAIN LOGIN\r\n
[PROTOCOL] [A380A9AE-4A27-49D8-944A-6A54C94C9D05.1] [queue/smtp_forward] S: 250 Ok\r\n
[PROTOCOL] [A380A9AE-4A27-49D8-944A-6A54C94C9D05.1] [queue/smtp_forward] C: MAIL FROM:<admin@test.com>
[PROTOCOL] [A380A9AE-4A27-49D8-944A-6A54C94C9D05.1] [queue/smtp_forward] S: 530 Must issue a STARTTLS command first\r\n
[INFO] [A380A9AE-4A27-49D8-944A-6A54C94C9D05.1] [core] hook=queue_outbound plugin=queue/smtp_forward function=hook_queue params="" retval=DENY msg="Must issue a STARTTLS command first"
[DEBUG] [A380A9AE-4A27-49D8-944A-6A54C94C9D05.1] [core] running deny hooks

Steps to reproduce

Here is my smtp_forward config:

host=email-smtp.us-west-2.amazonaws.com
port=587
enable_tls=true
auth_type=plain
auth_user=AKIAMSECRET
auth_pass= SECRETPASSWORD
max_connections=10

[Dexus]

First: I see a known bug: #1372 Second: Do you have activated the TLS Plugin? And set the config correct?

[BradRuderman]

@Dexus

1. Yes, I am on 2.8 alpha 7, I implemented the work-around with no luck
2. I have activated the TLS plugin, its working correctly for inbound. Meaning when a client connects to haraka, I don't receive any cert errors and am able to send mail. However when the smtp_forward plugin tries to send email, it is unable to. I am not sure why it would need a TLS configuration since its the client, however i did try all combinations of the new [outbound] in the tls.ini config with no luck.

[BradRuderman]

All my tls.ini config looks like:

[outbound]
ciphers=ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4
requestCert=false
rejectUnauthorized=false
;disable_for_failed_hosts=true

[inbound]
ciphers=ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4
;rejectUnauthorized=true
requestCert=true

2 issues:

1. For some reason when I uncomment rejectUnauthorized in inbound, I get a strange error, however when commented it works fine:

connection error: Error: 139946438494016:error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate:../deps/openssl/openssl/ssl/s3_srvr.c:3256:

1. When I use gmail as the smtp outbound server I also get the same error 530 Must issue a STARTTLS command first. It seems the only outbound server I can get working is sendgrid. Any ideas?

I am using node version 5.9.0 and patched based on https://github.com/haraka/Haraka/issues/1372

[BradRuderman]

Here is a screen shot of 2 different smtp relay servers. The first is amazon ses which is NOT working. The bottom is mailgun which does work.

[Dexus]

Look to the 250-Starttls and the 250 starttls...

Looks like s bug?

Von meinem iPhone gesendet

Am 19.03.2016 um 07:33 schrieb Brad Ruderman notifications@github.com:

Here is a screen shot of 2 different smtp relay servers. The first is amazon ses which is NOT working. The bottom is mailgun which does work.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub

[baudehlo]

SES requires TLS. Mailgun doesn't (though it's advisable anyway).

On Sat, Mar 19, 2016 at 4:39 AM, Josef Fröhle notifications@github.com wrote:

Look to the 250-Starttls and the 250 starttls...

Looks like s bug?

Von meinem iPhone gesendet

Am 19.03.2016 um 07:33 schrieb Brad Ruderman notifications@github.com:

Here is a screen shot of 2 different smtp relay servers. The first is amazon ses which is NOT working. The bottom is mailgun which does work.

— You are receiving this because you were mentioned.

Reply to this email directly or view it on GitHub

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/haraka/Haraka/issues/1387#issuecomment-198668328

[BradRuderman]

@baudehlo Do we think there is an issue with the smtp_client.js that it isn't recognizing 250-STARTTLS vs 250 STARTTLS?

Thanks!

[baudehlo]

That might be an issue - I didn't write smtp_client.js - we really should unify it with outbound.js which works just fine with multiple mixed capabilities - but reading the code - it looks like it doesn't cope well with STARTTLS in the capabilities in an earlier place.

On Sat, Mar 19, 2016 at 8:42 PM, Brad Ruderman notifications@github.com wrote:

@baudehlo https://github.com/baudehlo Do we think there is an issue with the smtp_client.js that it isn't recognizing 250-STARTTLS vs 250 STARTTLS?

Thanks!

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/haraka/Haraka/issues/1387#issuecomment-198816149

[BradRuderman]

@baudehlo Is the way to unify it to write additional plugins that use outbound.js instead of smtp_client.js for smtp_forward?

[baudehlo]

No. It's a bigger job than that. More internal work.

On Mar 19, 2016, at 8:51 PM, Brad Ruderman notifications@github.com wrote:

@baudehlo Is the way to unify it to write additional plugins that use outbound.js instead of smtp_client.js for smtp_forward?

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub

[BradRuderman]

@baudehlo On the roadmap? How can I help?

[baudehlo]

It has been on the roadmap for a while.

The basic plan is: migrate the SMTP sending code from outbound.js (which is much more capable) into smtp_client.js and use that code from outbound.js.

It's a fairly big task and problematic in some ways. If you want to take it on you are welcome but realize how much work it is. The two bits of code are currently very different.

On Mar 19, 2016, at 8:53 PM, Brad Ruderman notifications@github.com wrote:

@baudehlo On the roadmap? How can I help?

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub

[baudehlo]

The alternative is to copy the capabilities code from outbound and make it work in smtp_client.js - lots less work but not necessarily the right end goal. Honestly we would accept either as pull requests.

On Mar 19, 2016, at 8:53 PM, Brad Ruderman notifications@github.com wrote:

@baudehlo On the roadmap? How can I help?

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub

[BradRuderman]

@baudehlo Unfortunately I don't know the SMTP protocol well enough to build this. I think I might work around by building either an attempt to use outbound.js or simply an additional plugin which forwards mail to another smtp server.

[baudehlo]

It's easy to use outbound.js for this, for what it's worth - just use the get_mx hook to specify your inbound server. Just be careful you don't turn your server into an open relay.

On Sat, Mar 19, 2016 at 9:06 PM, Brad Ruderman notifications@github.com wrote:

@baudehlo https://github.com/baudehlo Unfortunately I don't know the SMTP protocol well enough to build this. I think I might work around by building either an attempt to use outbound.js or simply an additional plugin which forwards mail to another smtp server.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/haraka/Haraka/issues/1387#issuecomment-198817051

[baudehlo]

I'm wondering if this is related to #1392 - does the fix there need to also go into the forward plugin @msimerson ?

Also @smfreegard recently fixed a bug related to rejectUnauthorised - it might be worth checking that out. #1400

[msimerson]

I'm wondering if this is related to #1392 - does the fix there need to also go into the forward plugin @msimerson ?

Looks like no, since smtp_* plugins use smtp_client with already has a similar check.

[BradRuderman]

@baudehlo Is it possible to pass upstream failures with outbound.js back to the original client? For example if I am dynamically changing the relay destination with outbound js, I want to pass errors back to the client.

I guess after looking, I would have to do the logic to check if I need to bounce it in the queue_outbound hook instead of the get_mx hook. Then I could bounce it right away with a DENY code.

Thanks!

[baudehlo]

No it's not possible.

On Apr 3, 2016, at 7:51 PM, Brad Ruderman notifications@github.com wrote:

@baudehlo Is it possible to pass upstream failures with outbound.js back to the original client? For example if I am dynamically changing the relay destination with outbound js, I want to pass errors back to the client.

Thanks!

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub

[BradRuderman]

@baudehlo Is this the recommended implementation for smtp relay? Essentially I am building an smtp server that accepts emails and then relays them to different smtp servers do send (sendgrid, aws, etc). Currently I have hooked in the get_mx record and switch the mx based on the who it should go to, as determined by the auth user.

Thanks!

[baudehlo]

Sounds like a better way for you would be to modify smtp_forward or smtp_proxy to do what you need.

On Apr 3, 2016, at 8:44 PM, Brad Ruderman notifications@github.com wrote:

@baudehlo Is this the recommended implementation for smtp relay? Essentially I am building an smtp server that accepts emails and then relays them to different smtp servers do send (sendgrid, aws, etc). Currently I have hooked in the get_mx record and switch the mx based on the who it should go to, as determined by the auth user.

Thanks!

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub

[BradRuderman]

@baudehlo That was my original intention but the issues with TLS and the buggy smtp_client prevented that from working. :(

[baudehlo]

Make sure you're on the current alpha release and document those bugs then. We should probably close this issue and open a new one when you can do that.

On Apr 3, 2016, at 8:58 PM, Brad Ruderman notifications@github.com wrote:

@baudehlo That was my original intention but the issues with TLS and the buggy smtp_client prevented that from working. :(

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub

[BradRuderman]

@baudehlo I'm running the master branch. The initial ticket stands true to the bug I found. smtp_client is the culprit here as you mentioned.

That might be an issue - I didn't write smtp_client.js - we really should unify it with outbound.js which works just fine with multiple mixed capabilities - but reading the code - it looks like it doesn't cope well with STARTTLS in the capabilities in an earlier place.

[baudehlo]

I'd like @smfreegard to take a look at my assumptions there.

On Apr 3, 2016, at 9:25 PM, Brad Ruderman notifications@github.com wrote:

@baudehlo I'm running the master branch. The initial ticket stands true to the bug I found. smtp_client is the culprit here as you mentioned.

That might be an issue - I didn't write smtp_client.js - we really should unify it with outbound.js which works just fine with multiple mixed capabilities - but reading the code - it looks like it doesn't cope well with STARTTLS in the capabilities in an earlier place.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub

[BradRuderman]

@baudehlo @smfreegard Let me know how I can help here. Happy to work with you to solve this issue.

[corporatepiyush]

i am having the same issue with Node.js 4.2.2 and Haraka 2.7.3.

[PROTOCOL] [AA04B083-DE29-497E-A2FF-0FAE9E570C4A.1] [queue/smtp_forward] S: 250-smtp.gmail.com at your service, [106.221.156.48]\r\n [PROTOCOL] [AA04B083-DE29-497E-A2FF-0FAE9E570C4A.1] [queue/smtp_forward] S: 250-SIZE 35882577\r\n [PROTOCOL] [AA04B083-DE29-497E-A2FF-0FAE9E570C4A.1] [queue/smtp_forward] S: 250-8BITMIME\r\n [PROTOCOL] [AA04B083-DE29-497E-A2FF-0FAE9E570C4A.1] [queue/smtp_forward] S: 250-STARTTLS\r\n [PROTOCOL] [AA04B083-DE29-497E-A2FF-0FAE9E570C4A.1] [queue/smtp_forward] S: 250-ENHANCEDSTATUSCODES\r\n [PROTOCOL] [AA04B083-DE29-497E-A2FF-0FAE9E570C4A.1] [queue/smtp_forward] S: 250-PIPELINING\r\n [PROTOCOL] [AA04B083-DE29-497E-A2FF-0FAE9E570C4A.1] [queue/smtp_forward] S: 250-CHUNKING\r\n [PROTOCOL] [AA04B083-DE29-497E-A2FF-0FAE9E570C4A.1] [queue/smtp_forward] S: 250 SMTPUTF8\r\n [PROTOCOL] [AA04B083-DE29-497E-A2FF-0FAE9E570C4A.1] [queue/smtp_forward] C: MAIL FROM:<corporate.piyush@gmail.com> [PROTOCOL] [AA04B083-DE29-497E-A2FF-0FAE9E570C4A.1] [queue/smtp_forward] S: 530 5.7.0 Must issue a STARTTLS command first. zp5sm37352809pac.9 - gsmtp\r\n [INFO] [AA04B083-DE29-497E-A2FF-0FAE9E570C4A.1] [core] hook=queue_outbound plugin=queue/smtp_forward function=hook_queue params="" retval=DENY msg="5.7.0 Must issue a STARTTLS command first. zp5sm37352809pac.9 - gsmtp"

[BradRuderman]

@corporatepiyush FYI I created my own plugin that does this same thing using nodemailer/smtp-connection. It works.

Here it is: https://gist.github.com/BradRuderman/efccd46674641d02170663638950ee01

FYI make sure the connection.notes.relay is configured with : { smtp_host :"abc",smtp_port :123, smtp_username:'user', smtp_password:'abc'}