Mandrill encoding errors

[rymohr]

We use Mandrill to deliver our inbound emails and occasionally we'll get an encoding error with one of the email replies (expecting UTF-8 but get ASCII-8BIT).

Originally I thought it was an issue with Mandrill but they claim everything is converted to UTF-8 before being sent to the webhook. If they're converting the content to UTF-8 but leaving the original headers alone, wouldn't that screw up the decoding?

Here's what we're using to grab the body of the email:

def get_body(mail)
  if mail.multipart?
    if mail.text_part
      mail.text_part.decoded
    else
      mail.html_part.decoded
    end
  else
    mail.body.decoded
  end
end

Not sure if this is an issue with the mandrill strategy or just the way we're extracting the body of the message.

[joshuap]

We use the raw_msg key to parse each inbound message from their WebHook using ruby's standard Mail library. As long as they are converting their whole JSON payload to UTF-8, I'm not sure why you would get encoding errors. The Mail lib does some of its own encoding/decoding as I recall, though. Do you have an error message/backtrace for the conversion errors you mentioned?

[rymohr]

I do have a backtrace but it's not much help here since it doesn't blow up until I run the content through EmailReplyFilter. Just your typical incompatible character encodings: ASCII-8BIT and UTF-8 error.

Still waiting to hear back from Mandrill but I looked into one of the webhooks and it looks like only the event.msg.text field is converted to UTF-8. The content of raw_msg is in the original encoding (though encoded to be UTF-8 friendly I guess?).

We've been running into this error long enough that I finally just dropped incoming yesterday in favor of handling the webhooks directly and grabbing the body from event.msg.text. Seems to be working fine so far.

[joshuap]

That sounds about right (event.msg.text). We probably should do something to handle bad encodings in the raw_msg field or switch to use the UTF-8 keys. I used raw_msg because it's a lot simpler to construct a Mail::Message from. It sounds like you have this resolved on your end for now, but I'll leave this open until I can work up a fix. :)

[rymohr]

I wish I had saved some of my notes for you. It's a hard one to track down!

[rymohr]

This thread was a big help: https://github.com/mikel/mail/issues/431