Encoded headers violate RFC 2047

[skyler]

Here's a small Golang program which demonstrates the problem:

package main

import (
    "gopkg.in/gomail.v2"
    "os"
)

func main() {
    m := gomail.NewMessage()
    m.SetHeader("From", "skyler@sharpspring.com")
    m.SetHeader("To", "skyler@sharpspring.com")
    m.SetHeader("Subject", "{$firstname} Bienvendio a Apostólica, aquí inicia el camino de tu")
    m.SetBody("text/plain", "Hello, World")
    m.WriteTo(os.Stdout)
}

This program will output these headers:

Mime-Version: 1.0
Date: Wed, 16 Mar 2016 09:08:29 -0400
From: skyler@sharpspring.com
To: skyler@sharpspring.com
Subject: =?UTF-8?q?{$firstname}_Bienvendio_a_Apost=C3=B3lica,_aqu=C3=AD_inicia_el_camino_de_tu?=
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Note that the Subject header line is 96 characters long. According to RFC 2047,

While there is no limit to the length of a multiple-line header field, each line of a header field that contains one or more 'encoded-word's is limited to 76 characters.

The Subject header should have been folded at 76 characters but it was not.

[alexcesaro]

Did you run into an issue sending this email? Because even popular service like Gmail don't always respect these kind of rules.

Anyway, for now you can use Go 1.6 where encoded-words are automatically split which will reduce the line length. And I will soon fix that issue in Gomail.

[skyler]

The only issue I've seen is sending an email which is parsed on the receiving end by Python's email.message. I'm not 100% certain on this yet, but I think Python's email.message attempts to fold the lines and it folds them incorrectly (it does not specify the encoding for each line). If the header lines were properly folded then Python would not try to fold them itself, and would not break them. So not really Gomail's problem, but either way the behavior is not RFC compliant.

[alexcesaro]

This should be better. Please tell me if it works now. There is a little bug in the standard library which makes encoded-words a bit too long. I will fix it for Go 1.7.

[skyler]

I just got around to testing this. I've noticed two problems:

1. There's a newline inserted before the header content start
2. Long headers aren't folded :(

Here's my code:

package main

import (
    "gopkg.in/gomail.v2"
    "os"
)

func main() {
    var long_ss_header string = `{"id":"13892818549","companyName":"St-Jérôme","title":"Minister","firstName":"Levi","lastName":"Strauss","street":"8712 E. Jefferson Blvd","city":"Gainesville","country":"","state":"FL","zipcode":"32601","emailAddress":"lstrauss@myfavoriteisp.com","website":"http:\/\/www.baloneysandwich.com\/","phoneNumber":"999 012341239","officePhoneNumber":"","phoneNumberExtension":"","mobilePhoneNumber":"","faxNumber":"352 867-5309","description":"","numBounces":"0","hardBounced":"0","leadScore":"5","industry":"","isQualified":"","isContact":"0","hasOpportunity":"0","isUnsubscribed":"","emailaddress":"lstrauss@baloneysandwich.com","firstname":"Levi","lastname":"Strauss","companyname":"St-Jérôme","LeadID":"MzS3sDSuMLSwNDEFAA","EmailID":"M8MwMmAxNJEHAA","leadOwner":"","leadOwnerPhone":"","leadOwnerEmail":"","EmailAddress":"lstrauss@baloneysandwich.com","FirstName":"Levi","LastName":"Strauss","CompanyName":"St-Jérôme"}`
    var mergevars string = long_ss_header

    m := gomail.NewMessage()
    m.SetHeader("From", "skyler@sharpspring.com")
    m.SetHeader("To", "skyler@sharpspring.com")
    m.SetHeader("Subject", "${firstname} Bienvendio a Apostólica, aquí inicia el camino de tu")
    m.SetHeader("X-SP-MergeVars", mergevars)
    m.WriteTo(os.Stdout)
}

And here's the output:

Mime-Version: 1.0
Date: Mon, 28 Mar 2016 14:25:12 +0000
X-SP-MergeVars: 
 =?UTF-8?q?{"id":"13892818549","companyName":"St-J=C3=A9r=C3=B4me","title":"Minister","firstName":"Levi","lastName":"Strauss","street":"8712_E._Jefferson_Blvd","city":"Gainesville","country":"","state":"FL","zipcode":"32601","emailAddress":"lstrauss@myfavoriteisp.com","website":"http:\/\/www.baloneysandwich.com\/","phoneNumber":"999_012341239","officePhoneNumber":"","phoneNumberExtension":"","mobilePhoneNumber":"","faxNumber":"352_867-5309","description":"","numBounces":"0","hardBounced":"0","leadScore":"5","industry":"","isQualified":"","isContact":"0","hasOpportunity":"0","isUnsubscribed":"","emailaddress":"lstrauss@baloneysandwich.com","firstname":"Levi","lastname":"Strauss","companyname":"St-J=C3=A9r=C3=B4me","LeadID":"MzS3sDSuMLSwNDEFAA","EmailID":"M8MwMmAxNJEHAA","leadOwner":"","leadOwnerPhone":"","leadOwnerEmail":"","EmailAddress":"lstrauss@baloneysandwich.com","FirstName":"Levi","LastName":"Strauss","CompanyName":"St-J=C3=A9r=C3=B4me"}?=
From: skyler@sharpspring.com
To: skyler@sharpspring.com
Subject: 
 =?UTF-8?q?${firstname}_Bienvendio_a_Apost=C3=B3lica,_aqu=C3=AD_inicia_el_camino_de_tu?=

[skyler]

Also, I'd like to suggest two different functions (or options) for setting a header: the first would be a way to set a header that would not be modified in any way and the second would be a way to set a header that would be automatically encoded and folded as necessary.

[alexcesaro]

As I said earlier you need Go 1.6 to have the headers split. The headers will still be a bit too long but I sent a fix that should be in Go 1.7: https://go-review.googlesource.com/#/c/20918/

Concerning the newline at the start of the header, it doesn't seem forbidden by the RFCs but it might cause bugs in some client. Did you notice issues because of it?

[skyler]

Concerning the newline at the start of the header, it doesn't seem forbidden by the RFCs but it might cause bugs in some client. Did you notice issues because of it?

Not yet, but as I'm using this library to send email to all sorts of recipients with all sorts of different MUAs, I can't state with confidence that it would or would not cause an issue.

[skyler]

I tested with Go 1.6. Regarding the newline, I found an issue reading one of these headers using Python's email.message.Message. This is an example of a real email that I send using gomail, which is later read by Python:

Mime-Version: 1.0
Date: Wed, 30 Mar 2016 16:16:11 +0000
From: skyler@sharpspring.com
To: skyler@sharpspring.com
Subject: Hello, world! Today is Wednesday.Hello, world! Today is Wednesday.
 Hello, world! Today is Wednesday. Hello, world! Today is Wednesday.
X-SP-URLAppendQS:
 utm_medium=email&sslid=$LeadID&sseid=$EmailID&jobid=07532b9a-c62e-4383-96b6-1f982edb62cd

When this message is read in Python, an extra space is included before the start of the real header value:

In [67]: print s
Mime-Version: 1.0
Date: Wed, 30 Mar 2016 16:16:11 +0000
From: skyler@sharpspring.com
To: skyler@sharpspring.com
Subject: Hello, world! Today is Wednesday.Hello, world! Today is Wednesday.
 Hello, world! Today is Wednesday. Hello, world! Today is Wednesday.
X-SP-URLAppendQS:
 utm_medium=email&sslid=$LeadID&sseid=$EmailID&jobid=07532b9a-c62e-4383-96b6-1f982edb62cd

In [68]: msg = email.message_from_string(s)

In [69]: msg['X-SP-URLAppendQS']
Out[69]: ' utm_medium=email&sslid=$LeadID&sseid=$EmailID&jobid=07532b9a-c62e-4383-96b6-1f982edb62cd'

The extra space breaks later processing of the header.

[alexcesaro]

I updated the code not to insert a newline as the first character. The drawback is that a header line can now be longer than 76 characters.

I won't do more for two reasons:

1. It is not possible to get the correct behavior using the standard library. So doing it would mean adding a lot of code to Gomail.
2. More importantly, Gmail has the same behavior, I tried sending a mail with a long subject and here is what I got:

Subject: =?UTF-8?B?w6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqQ==?=
    =?UTF-8?B?w6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOp?=
    =?UTF-8?B?w6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOp?=
    =?UTF-8?B?w6nDqcOpw6nDqcOpIMOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6k=?=
    =?UTF-8?B?w6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOp?=
    =?UTF-8?B?w6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOp?=
    =?UTF-8?B?w6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOp?=
    =?UTF-8?B?w6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOp?=
    =?UTF-8?B?w6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOp?=
    =?UTF-8?B?w6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOp?=
    =?UTF-8?B?w6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOp?=
    =?UTF-8?B?w6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOp?=
    =?UTF-8?B?w6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOp?=
    =?UTF-8?B?w6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqcOpw6nDqQ==?=

The first encoded-word is 84 characters-long and the first line is 93 characters-long. Yet I suppose Gmail has a pretty good deliverability rate :smile:

So I am sorry but I think it is the Python library you are using that should be fixed. I cannot do reasonably much more.