MessageBounced event, how to you know it's a hard or soft bounce?

[twisted1919]

Hello,

Looking at https://github.com/atech/postal/wiki/Webhook-Events-&-Payloads#message-bounces it shows the original message and the bounce message, but how do you tell if it's been a hard or soft bounce? Do we look in the payload object in the status column? if so, beside HardFail is there anything else, like SoftFail ? The documentation is rather confusing for this as it does not show the response actually contains a payload and a event key and so on, i had to find that by digging in the code but since i am not a ruby guy, that's all i could do. Could we maybe get the documentation updated with full examples of data?

Thank you.

[adamcooke]

HardFail and SoftFail are nothing to do with bounces. These will be triggered if the mail server that should accept the message is unavailable or doesn't accept the message. For hard fails, the mail server actively told us that the message cannot be delivered - mailbox doesn't exist etc... Soft fails are when the server is unavailable and we should try again.

A bounce is when a message is sent back after the original message has been fully accepted by the mail server.

[twisted1919]

A bounce is when a message is sent back after the original message has been fully accepted by the mail server.

Yeah, of course. But the email can return because the email couldn't be delivered after a few days, for whatever fatal reasons(i.e email was not sent after retried few times and being in the queue for x days), or because they hit an autoresponder message and that is returned. In that case you have to differentiate between the two, one is a hard bounce while the other is a soft one.

So basically you are not taking these into consideration right now, is this correct?

[adamcooke]

Mail servers shouldn't be accepting mail unless it can be delivered immediately (although some do - Exchange accepts messages when mailboxes don't exist, for example). You shouldn't, for example, see a situation where a message cannot be delivered after a few days. This would be a SoftFail followed by a HardFail and no bounces should be triggered at all because the message isn't accepted by the mail server.

You can receive a bounce even if the message has been delivered (i.e. an auto response) or if it hasn't been delivered (i.e. mailbox doesn't exist on Exchange). To determine which it is would require parsing the content of the bounce message which isn't something Postal does.

[twisted1919]

Mail servers shouldn't be accepting mail unless it can be delivered immediately (although some do - Exchange accepts messages when mailboxes don't exist, for example). That's not entirely true. Providers like yahoo for instance will accept your email even if the recipient does not exist. Providers like Sparkpost / Mailgun / Etc, do the same, they queue the message which can go out hours later if the queue is high on your ip pool.

You shouldn't, for example, see a situation where a message cannot be delivered after a few days. Except you do see this. Play with postfix for a while and you'll see what i am talking about.

My point is that right now postal only does half of the work for bounce processing and this should be highlighted somewhere because users might get in troubles for this exact reason.

I personally don't send via postal yet (though i believe it might be the best open source project for what it does), i am here because my customers pointed me here.

Best of luck, maybe you'll improve this in future. Thanks.

[spheron1]

As @adamcooke said, receiving mail servers should not be accepting emails which they cannot deliver. If they do, then they are misconfigured.

Yahoo correctly reject emails for non-existent accounts at SMTP time, they do not queue and then bounce the message. After the sending mail server sends the DATA command, Yahoo will reply with "554 delivery error: dd This user doesn't have a yahoo.com account". The message is never accepted.

As far as I know, Sparkpost and Maligun are sending mail servers similar to Postal and not receiving mail servers, so aren't relevant.

Perhaps you can clarify what behaviour you see with Postfix exactly and in what configuration? Our Postfix servers all behave correctly and reject emails to non-existent addresses at SMTP time rather than accepting messages and then generating bounce backs later (backscatter).

[twisted1919]

@spheron1 - i am in a bit of a hurry now, but your assumption of

Yahoo correctly reject emails for non-existent accounts at SMTP time, they do not queue and then bounce the message.

Is wrong as far as i can tell. The reasoning is simple, if they would tell you directly what account exists and what not, would open the doors for spammers to find out if the addresses they have are correct or not. This is the same reason why smtp servers don't allow the VRFY command.

As @adamcooke said, receiving mail servers should not be accepting emails which they cannot deliver. If they do, then they are misconfigured.

Yeah but you see, your postal app will deliver in the entire world and you cannot use this rationale. Yes, some servers don't comply to standards, but this is how the email world works and there might be valid users behind misconfigured servers that will be affected.

Anyway, hope the above will help somehow. Thanks.

[spheron1]

It is trivial to verify this behaviour - just try sending email to a non-existent Yahoo.com address and then look at your logs or the bounce message from your local/sending MTA - you will see something like this:

Jul 7 13:31:48 KSP810-MTA1 postfix/smtp[28630]: 5201D780060: to=<wnbrwnuirwnbunrw943n894n9bn3459bn3459@yahoo.com>, relay=mta5.am0.yahoodns.net[66.196.118.36]:25, delay=2.2, delays=0.01/0.02/0.78/1.4, dsn=5.0.0, status=bounced (host mta5.am0.yahoodns.net[66.196.118.36] said: 554 delivery error: dd This user doesn't have a yahoo.com account (wnbrwnuirwnbunrw943n894n9bn3459bn3459@yahoo.com) [0] - mta1284.mail.bf1.yahoo.com (in reply to end of DATA command))

This is standard behaviour across mail servers.

The main method to stop enumeration attacks is by rate limiting connections from IP addresses where error responses are being generated. The more error responses and the less OK responses you send back, the longer you rate limit that IP address. Legitimate mail servers are not significantly affected by this as they will back off and re-retry genuine messages later, but it makes enumeration attacks infeasibly slow given the size of the search space.

Also, whether a message is rejected at SMTP time or accepted and then bounced doesn't particularly matter to a spammer trying to check for valid email addresses. It is trivial to send emails from unique addresses and then check for bounces to see which addresses are valid.

I'm not sure what;

there might be valid users behind misconfigured servers that will be affected.

is in regards to as you're asking about changing bounce handling behaviour to benefit the sender and so this doesn't affect deliverability in any way?

I don't represent the Postal developers in any form, but I would say that if you want the current behaviour changing then you need to demonstrate a reasonable use case. As it stands you haven't actually been able to provide an example of when messages are commonly accepted then later bounced and so this is at best to deal with rare corner cases and so would hardly a priority.

[premhunt]

I am agree with twisted1919 here.In practical scenario, you can not look only on per IP rate as impacted delayed message. There are multiple reasons are there for same and as IP or domain blacklist, the content of email, The mailbox are temporarily full etc.The 4XX series message should define a dead time treat as Soft. We can not treat these as a Hardbounce or left stuck in queue, we need to send information back to client in time.

[twisted1919]

It's fine, you'll understand at some point ;)

Sent from my iPhone

On 13 Jul 2017, at 18:01, Adam Cooke notifications@github.com wrote:

4xx are treated as soft.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[adamcooke]

Messages are handled differently based on the error code that is returned from Net::SMTP. If you feel like making a pull request to handle this differently please do. The current situation works fine for our use case and the majority of use cases so I have very little interest in looking handling this in any other way.

[twisted1919]

Unfortunately i don't use ruby so i wouldn't know how to even start nor i have the time to learn it.

I don't know what exactly you are handling and what not but for sure you're not covering this entirely.

Maybe someone with ruby knowledge can take onto this task and correct the way bounce handling works.

Thank you.

Sent from my iPhone

On 13 Jul 2017, at 19:16, Adam Cooke notifications@github.com wrote:

Messages are handled differently based on the error code that is returned from Net::SMTP. If you feel like making a pull request to handle this differently please do. The current situation works fine for our use case and the majority of use cases so I have very little interest in looking handling this in any other way.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[premhunt]

Let me give you one more example of Postal wrong softfail retry which is going since 13hr where as it should return HardFail and stop attempting.

I am looking on the code and will do pull request.

@twisted1919 I have done changes in existing webhook on postal side to support your MW bounce handler. It may not use here and let me know,if you required for any purpose.

[twisted1919]

I think this issue should be reopened till the bounce processing works as expected. @premhunt - I am not sure if you are aware, i am sorry if you are but i don't know your username in our community, but for now we have a webhook endpoint in MW which can handle what Postal is sending. If you're interested, i can share the implementation details with you, just pm me via forums.

[adamcooke]

If a domain cannot be resolved, that's a soft fail and it should be retried until the maximum retries has been reached (at which point will be a hard fail). This will not be changed.

A bounce is a bounce and nothing to do with a message soft failing or hard failing - that will not change. A message that has been accepted by a mail server, will always be marked as sent and any bounce will be treated as a new message and linked to the original.

I have no intention on reopening this issue I'm afraid. There is nothing suggested in this issue which is actionable.

[premhunt]

@adamcooke I hope, soon or later , you may need to look on these variables like Postfix for better delivery, better performance and more client application friendly respond. e.g of postfix - maximal_queue_lifetime = 1h maximal_backoff_time = 15m minimal_backoff_time = 5m queue_run_delay = 5m

So it can define, own setting to avoid multiple unnecessary request. Your application is very good with missing control on delivery option. If I will add, i will give a pull request.

@twisted1919 No, I am not aware of forum but sure heard of you from many on your MW and done some help with direct posifix usage.