search

KumoCorp / kumomta

The first Open-Source high-performance MTA developed from the ground-up for high-volume email sending environments.
https://kumomta.com
Apache License 2.0
249 stars 33 forks source link

HTML Breakage with Base64 Encoding and 'append_text_html' Function #243

Closed arunkvinam closed 3 months ago

arunkvinam commented 3 months ago

What Operating System are you seeing this problem on?

Rocky8

What Hardware is this system running?

Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 12 On-line CPU(s) list: 0-11 Thread(s) per core: 2 Core(s) per socket: 6 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel BIOS Vendor ID: Intel(R) Corporation CPU family: 6 Model: 158 Model name: Intel(R) Xeon(R) E-2286G CPU @ 4.00GHz BIOS Model name: Intel(R) Xeon(R) E-2286G CPU @ 4.00GHz Stepping: 10 CPU MHz: 4601.359 CPU max MHz: 4900.0000 CPU min MHz: 800.0000 BogoMIPS: 8016.00 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 12288K NUMA node0 CPU(s): 0-11

KumoMTA version

2024.05.23-11841789

Did you try the latest release to see if the issue is better (or worse!) than your current version?

Yes, and I updated the version box above to show the version that I tried

Describe the bug

I am trying to integrate an image pixel tracking link into my HTML email content using the following code:

local my_tracking_link = '<img src="http://10.0.0.1/img_tracker.jpg" alt="open tracking pixel">'
msg:append_text_html(my_tracking_link)

After appending this image pixel using the append_text_html function and encoding the email as base64, the HTML appears broken when viewed in a Gmail mailbox.

To Reproduce

No response

Configuration

local my_tracking_link = '<img src="http://10.0.0.1/img_tracker.jpg" alt="open tracking pixel">'
msg:append_text_html(my_tracking_link)

Expected Behavior

No response

Anything else?

No response

MHillyer commented 3 months ago

Pls provide the before and after HTML so it's clear how "the HTML appears broken".

arunkvinam commented 3 months ago

Before

<!DOCTYPE html>
                    <html>
                    <head>

                          <title>Sample HTML with Href Link</title>
                          </head>
                          <body>
                            <h1>Sample HTML with Href Link</h1>
                              <p>This is a sample HTML code with a hyperlink.</p>
                                <a href="https://www.example.com">Visit Example.com</a>
                                </body>
                                </html>
**After**
    <!DOCTYPE html>=0A                    <html>=0A                    <head>=
=0A                      =0A                          <title>Sample HTML wi=
th Href Link</title>=0A                          </head>=0A                =
          <body>=0A                            <h1>Sample HTML with Href Li=
nk</h1>=0A                              <p>This is a sample HTML code with =
a hyperlink.</p>=0A                                <a href=3D"https://www.e=
xample.com">Visit Example.com</a>=0A                               =20
<img src=3D"http://10.0.0.1/img_tracker.jpg" alt=3D"open tracking pixel"></=
body>=0A                                </html>=0A                         =
      =20

Note : use encoding - base64

Issue exist only using base64 encoding . there is no issue in normal encoding methods like 'quoted-printable' etc..

wez commented 3 months ago

Please share the smallest reproduction scenario for this; we need to see what is being injected (the complete message) and what is landing in the inbox (the complete message).

arunkvinam commented 3 months ago 
<?php

require 'PHPMailer/src/PHPMailer.php';
require 'PHPMailer/src/SMTP.php';
require 'PHPMailer/src/Exception.php';

use PHPMailer\PHPMailer\Exception;
use PHPMailer\PHPMailer\PHPMailer;
use PHPMailer\PHPMailer\SMTP;

try {
    $mail = new PHPMailer(true);
    $mail->SMTPDebug = SMTP::DEBUG_SERVER;
    $mail->isSMTP();
    $mail->Host = 'kumo.host.com';
    $mail->SMTPAuth = true;      
    $mail->Username =  'username';
    $mail->Password =   'password';      
    $mail->SMTPSecure = 'tls';
    $mail->AuthType = 'PLAIN';
    $mail->Port = 587;
    $mail->SMTPOptions = array(
                            'ssl' => array(
                                            'verify_peer' => false,
                                                    'verify_peer_name' => false,
                                                            'allow_self_signed' => true,
                                                                ),
                                                        );

    $mail->setFrom('info@domain.com', "Sender ");
    $mail->addAddress('email@gmail.com', "Recipient Name 1 Sure");
    $mail->Subject = "sample email";
    $mail->Encoding = 'base64';
    $mail->Sender = 'returnninfo@domain.com';
    $htmlContent = '<!DOCTYPE html>
                    <html>
                    <head>
                          <title>Sample HTML with Href Link</title>
                          </head>
                          <body>
                            <h1>Sample HTML with Href Link</h1>
                              <p>This is a sample HTML code with a hyperlink.</p>
                                <a href="https://www.example.com">Visit Example.com</a>
                                </body>
                                </html>
                                ';
    $mail->Body = $htmlContent;
    $mail->IsHTML(true);
    if (!$mail->send()) {
        echo 'Mailer Error: ' . $mail->ErrorInfo;
    } else {
        echo 'Message sent successfully!';
    }
} catch (Exception $e) {
    echo "Message could not be sent. Mailer Error: {$mail->ErrorInfo}";
}

Use above php code to sent emails and use append_text_html function in lua file

local my_tracking_link = '<img src="http://10.0.0.1/img_tracker.jpg" alt="open tracking pixel">'
msg:append_text_html(my_tracking_link)`

Issue exist in gmail mailbox

wez commented 3 months ago

We don't have time/resources to replicate your code and run it. Please share the complete message body produced by the code, before it is injected into kumomta, and the resulting complete message body as it appears in the mailbox.

arunkvinam commented 3 months ago

Before injecting image pixel using kumomta

 DKIM-Signature: v=1; a=rsa-sha256; d=domain.com; s=kumomta; c=relaxed/relaxed;
         bh=GvTClDQcCi5fNfwzYxQElOfPSJ1EfVLmbGXKh6WR7Ac=;
         h=from:to:subject; t=1722229496;
         b=LZSDnaCSK5ghfeObR3NdyyJRPOHE+Fnt1+cTzSa9EYn6Qmzh7r4Mc7uiwbDMFi67f7/jnsnlk
         kUh0cuVrxVDirb4TR6TUOkvrZ8cTNS0Q6vCGH00PtPAC0pgHTH8WelC07xRD00kMA+aVe/LFysP
         1qM04nOJCB1yMzAVg6jMDRY=;
 DKIM-Signature: v=1; a=rsa-sha256; d=domain.com; s=kmdk; c=relaxed/relaxed;
         bh=GvTClDQcCi5fNfwzYxQElOfPSJ1EfVLmbGXKh6WR7Ac=;
         h=from:to:subject; t=1722229496;
         b=FSxEqP4Uu+NhW8h/VD9+hHS1ogIKJmrNL3CBIceneXZW3gIiLRHXw/mYlLaCFFGGIUkjiBHIt
         gLEz/ju1k98kjMF1UY0SFABJ9krnW8TMmBfVdRPjnXzrwxdm0TJ0p3vW5Db+NendsilCpTvp1T+
         cakR5Sabk58iO4e6mIhLlD9Gu97T+025EFDwlhtZ0yKuDpSyEWLYwCmUf4QUK8BVlIlyXJAj8lB
         OOV7/J/7FzsmRa9Mx/o1pTXYH+AZiEgB4uwbFI78PqN7QMtpUEZxWhKEBpjfN3suMBHEY42me1G
         GnYDjvATEI2buDrMrF5BR+jMV3JWc/6Om/rvIdGPhiaA==;
 Received: from development.domain.com (10.0.8.1)
   by kumo.domain.com (KumoMTA 10.0.8.1)
   with ESMTP id 188049cc4d6811efac760cc47a84106e for <arunkvinamcalicut@domain.com>;
   Mon, 29 Jul 2024 05:04:55 +0000
 Date: Mon, 29 Jul 2024 05:04:55 +0000
 To: Recipient Name 1 Sure <***********@domain.com>, Recipient Name 1 Sure <arunk*****@gmail.com>, Recipient Name 1 <*********@gmail.com.com>
 From: Sender <aruninfo@domain.com>
 Subject: Arun Transaction open check mail(base64) - 1 with inject
 X-Mailer: PHPMailer 6.9.1 (https://github.com/PHPMailer/PHPMailer)
 mailercloud-track-opens: true
 MIME-Version: 1.0
 Content-Type: text/html; charset=iso-8859-1
 Content-Transfer-Encoding: base64
 Message-ID: <188049cc4d6811efac760cc47a84106e@domain.com>
 X-Tracking-Id: 188049cc4d6811efac760cc47a84106e
 X-SentFromServer: 10.0.8.1:58816_18_19
 X-campaign-Id: 422_domain.com
 [1B blob data]
 PCFET0NUWVBFIGh0bWw+CiAgICAgICAgICAgICAgICAgICAgPGh0bWw+CiAgICAgICAgICAgICAg
 ICAgICAgPGhlYWQ+CiAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAgICAg
 ICAgICA8dGl0bGU+U2FtcGxlIEhUTUwgd2l0aCBIcmVmIExpbms8L3RpdGxlPgogICAgICAgICAg
 ICAgICAgICAgICAgICAgIDwvaGVhZD4KICAgICAgICAgICAgICAgICAgICAgICAgICA8Ym9keT4K
 ICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxoMT5TYW1wbGUgSFRNTCB3aXRoIEhyZWYgTGlu
 azwvaDE+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxwPlRoaXMgaXMgYSBzYW1wbGUg
 SFRNTCBjb2RlIHdpdGggYSBoeXBlcmxpbmsuPC9wPgogICAgICAgICAgICAgICAgICAgICAgICAg
 ICAgICAgIDxhIGhyZWY9Imh0dHBzOi8vd3d3LmV4YW1wbGUuY29tIj5WaXNpdCBFeGFtcGxlLmNv
 bTwvYT4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8L2JvZHk+CiAgICAgICAgICAg
 ICAgICAgICAgICAgICAgICAgICAgPC9odG1sPgogICAgICAgICAgICAgICAgICAgICAgICAgICAg
 ICAgIA==
 [1B blob data]
arunkvinam commented 3 months ago

After injecting image pixel using kumomta

 DKIM-Signature: v=1; a=rsa-sha256; d=domain.com; s=kumomta; c=relaxed/relaxed;
         bh=GvTClDQcCi5fNfwzYxQElOfPSJ1EfVLmbGXKh6WR7Ac=;
         h=from:to:subject; t=1722229496;
         b=LZSDnaCSK5ghfeObR3NdyyJRPOHE+Fnt1+cTzSa9EYn6Qmzh7r4Mc7uiwbDMFi67f7/jnsnlk
         kUh0cuVrxVDirb4TR6TUOkvrZ8cTNS0Q6vCGH00PtPAC0pgHTH8WelC07xRD00kMA+aVe/LFysP
         1qM04nOJCB1yMzAVg6jMDRY=;
 DKIM-Signature: v=1; a=rsa-sha256; d=domain.com; s=kmdk; c=relaxed/relaxed;
         bh=GvTClDQcCi5fNfwzYxQElOfPSJ1EfVLmbGXKh6WR7Ac=;
         h=from:to:subject; t=1722229496;
         b=FSxEqP4Uu+NhW8h/VD9+hHS1ogIKJmrNL3CBIceneXZW3gIiLRHXw/mYlLaCFFGGIUkjiBHIt
         gLEz/ju1k98kjMF1UY0SFABJ9krnW8TMmBfVdRPjnXzrwxdm0TJ0p3vW5Db+NendsilCpTvp1T+
         cakR5Sabk58iO4e6mIhLlD9Gu97T+025EFDwlhtZ0yKuDpSyEWLYwCmUf4QUK8BVlIlyXJAj8lB
         OOV7/J/7FzsmRa9Mx/o1pTXYH+AZiEgB4uwbFI78PqN7QMtpUEZxWhKEBpjfN3suMBHEY42me1G
         GnYDjvATEI2buDrMrF5BR+jMV3JWc/6Om/rvIdGPhiaA==;
 Received: from development.domain.com (10.0.8.1)
   by kumo.domain.com (KumoMTA 10.0.8.1)
   with ESMTP id 188049cc4d6811efac760cc47a84106e for <arunkvinamcalicut@domain.com>;
   Mon, 29 Jul 2024 05:04:55 +0000
 Date: Mon, 29 Jul 2024 05:04:55 +0000
 To: Recipient Name 1 Sure <arunkvinamcalicut@domain.com>, Recipient Name 1 Sure <arunkvinam@gmail.com>, Recipient Name 1 <arunk@vinamsolutions.com>
 From: Sender <aruninfo@domain.com>
 Subject: Arun Transaction open check mail(base64) - 1 with inject
 X-Mailer: PHPMailer 6.9.1 (https://github.com/PHPMailer/PHPMailer)
 mailercloud-track-opens: true
 MIME-Version: 1.0
 Content-Transfer-Encoding: base64
 Message-ID: <188049cc4d6811efac760cc47a84106e@domain.com>
 X-Tracking-Id: 188049cc4d6811efac760cc47a84106e
 X-SentFromServer: 10.0.8.1:58816_18_19
 X-campaign-Id: 422_domain.com
 Content-Type: text/html;
         charset="us-ascii"
 Content-Transfer-Encoding: quoted-printable
 [1B blob data]
 <!DOCTYPE html>=0A                    <html>=0A                    <head>=
 =0A                      =0A                          <title>Sample HTML wi=
 th Href Link</title>=0A                          </head>=0A                =
           <body>=0A                            <h1>Sample HTML with Href Li=
 nk</h1>=0A                              <p>This is a sample HTML code with =
 a hyperlink.</p>=0A                                <a href=3D"https://www.e=
 xample.com">Visit Example.com</a>=0A                               =20
 <img src=3D"http://arungotracking.domain.com/tr-opens/MTg4MDQ5Y2M0ZDY4MTF=
 lZmFjNzYwY2M0N2E4NDEwNmVfNDIy/blank.gif" style=3D"display:none !important;"=
   alt=3D"open tracking pixel"></body>=0A                                </h=
 tml>=0A                               =20
wez commented 3 months ago

the content itself looks fine to me, but I noticed that you are signing before modifying the content; that will invalidate the dkim signature. Always sign the message as a the last step!

arunkvinam commented 3 months ago

We are facing a critical issue affecting our email delivery functionality. Despite previous communications, the problem persists and is unrelated to DKIM configurations as initially suspected.

Key points:

1.  HTML Content Breakage: When emails are sent to Gmail mailboxes, the HTML formatting is disrupted, making the content unusable.
2.  append_text_html Function: This disruption coincides with our use of the append_text_html function in our system.
3.  Encoding Conversion: We use base64 encoding for our emails, but when using the append_text_html function, the encoding changes to quoted-printable. This change likely causes the HTML to break.
4.  Impact: This issue significantly affects our ability to use Kumomta for sending emails, impacting our operational capabilities and the reliability of our communications.

Additional concerns:

1.  Base64 to HTML Encoding: When we send an email with base64 encoding and enable open tracking, the encoding changes to HTML. This is a major concern for us.
2.  Gmail HTML Breakage: The output HTML is broken in Gmail mailboxes.

This is a severe issue requiring immediate attention.

wez commented 3 months ago

Your input message decodes like this:

; base64 -d
PCFET0NUWVBFIGh0bWw+CiAgICAgICAgICAgICAgICAgICAgPGh0bWw+CiAgICAgICAgICAgICAg
ICAgICAgPGhlYWQ+CiAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAgICAg
ICAgICA8dGl0bGU+U2FtcGxlIEhUTUwgd2l0aCBIcmVmIExpbms8L3RpdGxlPgogICAgICAgICAg
ICAgICAgICAgICAgICAgIDwvaGVhZD4KICAgICAgICAgICAgICAgICAgICAgICAgICA8Ym9keT4K
ICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxoMT5TYW1wbGUgSFRNTCB3aXRoIEhyZWYgTGlu
azwvaDE+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxwPlRoaXMgaXMgYSBzYW1wbGUg
SFRNTCBjb2RlIHdpdGggYSBoeXBlcmxpbmsuPC9wPgogICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgIDxhIGhyZWY9Imh0dHBzOi8vd3d3LmV4YW1wbGUuY29tIj5WaXNpdCBFeGFtcGxlLmNv
bTwvYT4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8L2JvZHk+CiAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgPC9odG1sPgogICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgIA==
<!DOCTYPE html>
                    <html>
                    <head>

                          <title>Sample HTML with Href Link</title>
                          </head>
                          <body>
                            <h1>Sample HTML with Href Link</h1>
                              <p>This is a sample HTML code with a hyperlink.</p>
                                <a href="https://www.example.com">Visit Example.com</a>
                                </body>
                                </html>

The output you shared is consistent with a quoted-printable rendition of that same input, with the text you specified inserted before the </body> tag.

The functionality appears to be working as described by the documentation.

What's missing from your report is a more detailed description of the "breakage". What is the practical impact? Are you just concerned that the quoted-printable encoding looks weird? Is that a practical impact to how it renders in the gmail web interface?

Without those details, we are left to assume that the issue is that you are misunderstanding how MIME encoding works.

arunkvinam commented 3 months ago

Thank you for your input. Despite the changes, we are still facing issues with HTML content breakage when emails are sent to Gmail mailboxes.

Key points:

1.  HTML Breakage: Emails sent to Gmail have disrupted HTML formatting, rendering the content unusable.
2.  append_text_html Function: This issue coincides with using the append_text_html function.
3.  Encoding Conversion: Using the append_text_html function changes the encoding from base64 to quoted-printable, likely causing the HTML breakage.

This issue significantly impacts our email delivery and operational capabilities. We urgently need a solution.

MHillyer commented 3 months ago

The output you shared is consistent with a quoted-printable rendition of that same input, with the text you specified inserted before the tag.

The functionality appears to be working as described by the documentation.

What's missing from your report is a more detailed description of the "breakage". What is the practical impact? Are you just concerned that the quoted-printable encoding looks weird? Is that a practical impact to how it renders in the gmail web interface?

Without those details, we are left to assume that the issue is that you are misunderstanding how MIME encoding works.

arunkvinam commented 3 months ago

Hi @MHillyer

Thank you for your response. As a developer, I’m encoding the email in base64 and expect the final header in the email client to remain in base64. This is a standard practice for developers using systems like Kumomta. It’s important that the encoding is not changed by the MTA, as consistency in encoding is crucial for maintaining the integrity of the email content across different clients.

I appreciate your attention to this matter and look forward to a resolution.

wez commented 3 months ago

The function you are using operates by decoding the part, updating the data, and then using an implementation defined choice in selecting the encoding to be used for the re-encoded part. There is no requirement to encode with the original transfer encoding because that encoding is, by definition, independent of the decoded content, and has no bearing on how that content will be consumed or displayed.

Please read RFC 2045 Section 6.2 for more information on the semantics of transfer encoding.

The resolution is that there is nothing to fix here; everything is operating as intended and documented.

arunkvinam commented 3 months ago

@wez Thanks for the update. However, the HTML appears broken when the mail reaches the Gmail mailbox. Please check the HTML data below. This is the HTML that is received in the Gmail mailbox, which is the main issue I am facing:

PCFET0NUWVBFIGh0bWw+PTBBICAgICAgICAgICAgICAgICAgICA8aHRtbD49MEEgICAgICAgICAg ICAgICAgICAgIDxoZWFkPj0NCj0wQSAgICAgICAgICAgICAgICAgICAgICA9MEEgICAgICAgICAg ICAgICAgICAgICAgICAgIDx0aXRsZT5TYW1wbGUgSFRNTCB3aT0NCnRoIEhyZWYgTGluazwvdGl0 bGU+PTBBICAgICAgICAgICAgICAgICAgICAgICAgICA8L2hlYWQ+PTBBICAgICAgICAgICAgICAg ID0NCiAgICAgICAgICA8Ym9keT49MEEgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGgxPlNh bXBsZSBIVE1MIHdpdGggSHJlZiBMaT0NCm5rPC9oMT49MEEgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICA8cD5UaGlzIGlzIGEgc2FtcGxlIEhUTUwgY29kZSB3aXRoID0NCmEgaHlwZXJsaW5r LjwvcD49MEEgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxhIGhyZWY9M0QiaHR0cHM6 Ly93d3cuZT0NCnhhbXBsZS5jb20iPlZpc2l0IEV4YW1wbGUuY29tPC9hPj0wQSAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICA9MjANCjxpbWcgc3JjPTNEImh0dHA6Ly9hcnVuZ290cmFja2lu Zy5zYWxlc2ppby5jb20vdHItb3BlbnMvWkdWaVlUWTJNakUwWkRreE1URj0NCmxabUkzTkRVd1ky TTBOMkU0TkRFd05tVmZOREl5L2JsYW5rLmdpZiIgc3R5bGU9M0QiZGlzcGxheTpub25lICFpbXBv cnRhbnQ7Ij0NCiAgYWx0PTNEIm9wZW4gdHJhY2tpbmcgcGl4ZWwiPjwvYm9keT49MEEgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIDwvaD0NCnRtbD49MEEgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgPTIwDQo=

`<!DOCTYPE html>=0A =0A = =0A =0A Sample HTML wi= th Href Link=0A =0A =

=0A

Sample HTML with Href Li= nk

=0A

This is a sample HTML code with = a hyperlink.

=0A Visit Example.com=0A =20 =0A =0A =20 ` My original HTML is below : Sample HTML with Href Link

Sample HTML with Href Link with php

This is a sample HTML code with a hyperlink.

Visit Example.com the HTML that is received in the Gmail mailbox and my original html is different.