Quote and Apostrophe characters within email communications are received as straight quote marks instead of their curved version

BEGamache commented 5 years ago

Description

Emails sent using the Email Wizard encodes the email a little differently than the Simplified version. Our issue centers around quotes and apostrophes (not certain if any other characters or marks are affected). When sent out using the Email Wizard they appear as straight quotes or apostrophes (or inch and foot marks). When sent using the Simplified version the marks are curved. It may not even be an issue with the style of editor used but a script(s) may be converting these characters back to plain text?

When using the WYSIWYG Email Wizard editor the marks are displayed as curved but don't remain that way when the email is received.

For testing, I used the Blank Wizard template and the Blank Simplified template. I have also tried a few of our custom templates. I have typed in the characters (this results in the straight versions), copied the marks from Word or another text editor so that I know I am copying the curved version, and also used their HTML code equivalents: “ ” ‘ ’

My original thought was the Content-Transfer-Encoding was getting set to 7bit when I received emails sent from the Email Wizard editor. I have tested two other email transports and got different results--however, the characters remained straight instead of curved.

Email Wizard (Mandrill):

Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 7bit

Simplified (Mandrill):

Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Simplified SMTP Sendgrid:

Content-Type: text/html; charset=us-ascii
Content-Transfer-Encoding: quoted-printable

Simplified Wizard (Mailgun):

Content-Type: text/html; charset="ascii"
Mime-Version: 1.0
Content-Transfer-Encoding: quoted-printable

I also found when I used the HTML character code entities and left the source code editor visible then sent a test email, the characters would be received correctly because they remained in their HTML character coding. Clicking Send Test would cause the source code editor to switch back to the WYSIWYG editor. When another subsequent send test is sent after this switch the received email will have the characters back as straight. For my custom templates, I am using Lava code that the WYSIWYG editor doesn't like so it keeps the email in the source code editor.

Could there be a JS or some other coding that is switching these special characters to plain text? I know we can't use < or > in lava conditions because they are changed to > and <, which makes me wonder if something similar is happening to quotes and apostrophes.

I believe I am using the correct meta tags in my templates but not getting the desired results:

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
        <meta charset="utf-8">
        <meta name="viewport" content="width=device-width, initial-scale=1">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">
    </head>
    <body>

Steps to Reproduce

Create a new communication using the Wizard editor.
Use the blank template and drag a one column container onto the editor and then a text container into that.
Replace the Title wording by copying and pasting an apostrophe and a quote from Word or text editor so that the mark is curved, then add “ ” ‘ “ through the source editor.
The editor will display it correctly but send a test (or final) and the received email will contain all straight marks.
Create a new communication using the Simplified editor.
Using the blank template, copy and paste an apostrophe and a quote mark from Word or another text editor into the WYSIWYG editor then add “ ” ‘ “ by switching to the source editor.
The editor will display it correctly. Send a test (or final), with the source code editor visible, and the received email will display the marks properly.

Expected behavior:

Curved quotes and apostrophes are expected to remain curved when sent and received.

Actual behavior:

Curved quotes are changed to straight quotes when received.

Versions

Rock Version: 8.7
Client Culture Setting: en-US

ajoythomas commented 5 years ago

I noticed this same issue. Some more color commentary: Emojis and special characters work when emails are triggered via workflow, but it doesn't work when you are sending a standard email via communication (using either email wizard or simple email template). It looks fine when you are previewing it in RockRMS, but when you see it via your email client it is not rendered. I've attached what you see via preview and what you see when you see it via email client. The difference is in the Content-type. it needs to be utf-8, but it puts in as Content-Type: multipart/alternative;

BEGamache commented 5 years ago

Also noticed a similar issue with Word merge docs. The original word file will have the curved smart quotes but the exported merge doc reverts to straight quotes. Not certain if it is a part of the same issue or a separate one. The nice thing with Word is you have Find and Replace to correct the quotes.

We haven't tried emojis in email body but we have tried them in the subject line with success (we only tried the smiley face emojis) but that is interesting to know.

ajoythomas commented 5 years ago

I can't quite tell if this is an issue with RockRMS Communication side of things or if it is on the transport side of things. We are currently using mailgun and this is the response I got from mailgun:

We have been able to confirm that there is an encoding/decoding bug on our side that is causing the behavior you are experiencing with the stripped-html. We have created an internal bug report and our devs will need to further investigate and fix.

I confirmed it will not be a quick fix and I do not have an ETA of when a fix will be in place. As for a workaround, while this is being investigated, you may consider retrieving the full unparsed MIME using the message-url by passing the header "Accept: application/mime". Then you can try parsing the message on your end if you would like to do so. You can read more about retrieving stored messages in our documentation here: https://documentation.mailgun.com/en/latest/api-sending.html#retrieving-stored-messages

Sharing this to see if it can help someone to get to the root cause faster.

MichaelAllen commented 5 years ago

The replacement of smart quotes with straight quotes is likely related to #2818.

jonedmiston commented 1 year ago

@MichaelAllen is correct. This is a duplicate of the same issue https://github.com/SparkDevNetwork/Rock/issues/2818

SparkDevNetwork / Rock