Replace text in message content on base64 messages

iammattmartin commented 6 years ago

We've been testing and on base64 encoded messages, replacing text (to remove previous replies) does not appear to work.

It looks like the matching occurs before the content is processed to text.

I'd assume this might be a bug but may also be valid as the content is encoded.

jstanden commented 6 years ago

Hi Matt! I'll test this and get right back to you. I assume this is on the 'Before new incoming message is accepted' event, right?

iammattmartin commented 6 years ago

It is indeed.

I've tried to match on the smallest bit of text to minimise the risk that the string is not being matched but even just two words fails entirely.

jstanden commented 6 years ago

Do you have an example of one of these message sources, along with the behavior you're running?

I'm not able to reproduce the issue with a sample like:

From: customer@cerb.example
To: support@cerb.example
Subject: Base64 encoded mesage
Content-Type: text/plain
Content-Transfer-Encoding: base64

VGhpcyBpcyBhIHRlc3QgbWVzc2FnZSB0aGF0IHdhcyBCYXNlNjQtZW5jb2RlZA==

Where the decoded body is "This is a test message that was Base64-encoded".

I can set up a rule like:

And the ticket we parse becomes:

The transfer decoding from Base64 would be happening in ext/Mailparse, and it's not something we're doing at the app level.

The raw email message may give a clue -- like the content actually coming from a message/rfc822 attachment or something. Or being double base64-encoded by the sender (stranger things have happened). I wouldn't suspect the latter if you do see normal text in the ticket (just not substituted by your bot).

You could peek into what the bot sees for that message by having the bot send you a copy of the {{body}} placeholder (e.g. email, comment, notification). It shouldn't be base64-encoded.

jstanden commented 6 years ago

If you need to capture a raw message, you can use: https://cerb.ai/tips/archive-inbound-mail/

iammattmartin commented 6 years ago

I wonder if it is their headers in that case not setting something but the message is still being pushed through.

Only have the headers from the message we do have which are unlikely to help.

Content-Type: multipart/related;
    boundary="_005_f29c5d01460f4f8c80dce3e3226edaeb_";
    type="multipart/alternative"

Have enabled and will get a message checked once we get a new one which will likely be Monday.

jstanden commented 6 years ago

Yeah, so there's most likely a pair of text/plain and text/html within that multipart/related. Inside one (or both) of those we'd fine the Base64. I'll check back on Monday when you have the full message source.

iammattmartin commented 6 years ago

In the headers as before:

Content-Type: multipart/related;
        boundary="_005_2bc29b10919b47f5a2e5fcf9a2eb66f9DSYEXPR03_";
        type="multipart/alternative"

Then the message with content stripped.

--_005_2bc29b10919b47f5a2e5fcf9a2eb66f9DSYEXPR03_
Content-Type: multipart/alternative;
    boundary="_000_2bc29b10919b47f5a2e5fcf9a2eb66f9DSYEXPR03_"

--_000_2bc29b10919b47f5a2e5fcf9a2eb66f9DSYEXPR03_
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: base64

ENCODED MESSAGE
--_000_2bc29b10919b47f5a2e5fcf9a2eb66f9DSYEXPR03_
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: base64

ENCODED MESSAGE
--_000_2bc29b10919b47f5a2e5fcf9a2eb66f9DSYEXPR03_--

--_005_2bc29b10919b47f5a2e5fcf9a2eb66f9DSYEXPR03_
Content-Type: image/jpeg; name="~WRD209.jpg"
Content-Description: ~WRD209.jpg
Content-Disposition: inline; filename="~WRD209.jpg"; size=823;
        creation-date="Mon, 18 Jun 2018 10:41:23 GMT";
        modification-date="Mon, 18 Jun 2018 10:41:23 GMT"
Content-ID: <~WRD209.jpg>
Content-Transfer-Encoding: base64

ENCODED MESSAGE
--_005_2bc29b10919b47f5a2e5fcf9a2eb66f9DSYEXPR03_
Content-Type: image/jpeg; name="image001.jpg"
Content-Description: image001.jpg
Content-Disposition: inline; filename="image001.jpg"; size=38085;
        creation-date="Mon, 18 Jun 2018 10:42:40 GMT";
        modification-date="Mon, 18 Jun 2018 10:42:40 GMT"
Content-ID: <image001.jpg@01D406F9.7561BCF0>
Content-Transfer-Encoding: base64

ENCODED MESSAGE
--_005_2bc29b10919b47f5a2e5fcf9a2eb66f9DSYEXPR03_--

iammattmartin commented 6 years ago

@jstanden Do you know if this will be fixed for 8.3.10 as its causing a few issues with stripping messages and leaving lots of previous data from long reply threads. :(

jstanden commented 6 years ago

Hey Matt! I'm still not able to reproduce this on the latest 8.3.x version. Even with the above example, the behavior sees the decoded content.

You could encrypt and send the full raw message from: https://keybase.io/encrypt#wgm

iammattmartin commented 6 years ago

As requested.

jstanden commented 6 years ago

Hi Matt,

Thanks! I can reproduce the issue with that message. The text is replaced properly in the plaintext part, but not the HTML part.

iammattmartin commented 6 years ago

Phew. I'm not going entirely mad then ;)

jstanden commented 6 years ago

Fixed (finally!) in 9.0.4

iammattmartin commented 6 years ago

Thanks @jstanden :) 👍

jstanden commented 6 years ago

Sure. Sorry for the huge delay. :)

jstanden / cerb

Replace text in message content on base64 messages #679