search

kisli / vmime

VMime Mail Library
http://www.vmime.org
GNU General Public License v3.0
273 stars 110 forks source link

utf8-quoted-printable encoding of to-/from-fields #111

Closed SoulfreezerXP closed 9 years ago

SoulfreezerXP commented 9 years ago

Hello Vincent,

I have a question to the automatic "utf8-quoted-printable encodings" of the to- and from-header-fields of an E-Mail, which vmime performs.

I do not know, if the following example is really RFC-compliant, but here we built a raw plain-text mail from the scratch and inserted an UTF-8-Character "ö" in the middle of the to- and from-fields. Vmime is passing the to- and from-fields directly to the smtp-server. No explicit encoding/decoding is done. Maybe my building of the E-Mail is not RFC-compliant, (I do not know ...), but the gmail-server seems to be satisfied with that.

#define VMIME_STATIC                                                                                       
#include <vmime/vmime.hpp>                                                                                 
#include <iostream>                                                                                        

struct MyCV : vmime::security::cert::defaultCertificateVerifier                                            
{                                                                                                          
    void verify(__attribute__((unused))vmime::shared_ptr<vmime::security::cert::certificateChain> c,       
                __attribute__((unused))const vmime::string& h) {}                                          
};                                                                                                         

struct MyTF : vmime::net::tracerFactory                                                                    
{                                                                                                          
    struct MyT : vmime::net::tracer                                                                        
    {                                                                                                      
        MyT(const vmime::string& proto, const int cId) : m_p(proto), m_cId(cId) {}                         
        void traceSend(const vmime::string& l){traceReceive("(Client)" + l);}                              
        void traceReceive(const vmime::string& l){std::cout << m_p << ":" << m_cId << "->" << l << std::end
        const vmime::string m_p;                                                                           
        const int m_cId;                                                                                   
    };                                                                                                     

    vmime::shared_ptr<vmime::net::tracer> create(vmime::shared_ptr<vmime::net::service> serv, const int cId
    {                                                                                                      
        return vmime::make_shared<MyT>(serv->getProtocolName(), cId);                                      
    }                                                                                                      
};                                                                                                         

int main()                                                                                                 
{                                                                                                          
    try                                                                                                    
    {                                                                                                      
        vmime::shared_ptr<vmime::net::session> sess = vmime::make_shared<vmime::net::session>();           
        sess->getProperties().setProperty("transport.protocol","smtps");                                   
        vmime::shared_ptr<vmime::net::transport> trans = sess->getTransport();                             
        trans->setProperty("server.address","smtp.gmail.com");                                             
        trans->setProperty("server.port",465);                                                             
        trans->setProperty("options.need-authentication", true);                                           
        trans->setProperty("auth.username", "vmime4711@gmail.com");                                        
        trans->setProperty("auth.password", "vmimetest");                                                  
        trans->setProperty("options.chunking",false);                                                      
        trans->setCertificateVerifier(vmime::make_shared<MyCV>());                                         
        trans->setTracerFactory(vmime::make_shared<MyTF>());                                               
        trans->connect();                                                                                  

        std::string strCRLF = "\r\n";                                                                      

        char strRFC2882date[64];                                                                           
        std::time_t result = std::time(nullptr);                                                           
        strftime(strRFC2882date, sizeof(strRFC2882date), "%a, %d %b %Y %T %z", std::localtime(&result));   

        std::stringstream ssUTF8MailPlain;                                                                 

        ssUTF8MailPlain << "Subject: Test"                               << strCRLF                        
                        << "From: \"vmime4711\" <vmimeö4711@gmail.com>" << strCRLF                        
                        << "To:  \"vmime4711\" <vmimeö4711@gmail.com>"  << strCRLF                        
                        << "Date: " << strRFC2882date                    << strCRLF                        
                        << "Mime-Version: 1.0"                           << strCRLF                        
                        << "Content-Type: text/plain; charset=utf-8"     << strCRLF                        
                        << "Content-Transfer-Encoding: quoted-printable" << strCRLF                        
                        << strCRLF                                                                         
                        << "Bodycontent";                                                                  

        vmime::shared_ptr<vmime::message> msg = vmime::make_shared<vmime::message>();                      
        msg->parse(ssUTF8MailPlain.str());                                                                 
        trans->send(msg);                                                                                  
        trans->disconnect();                                                                               
        return 0;                                                                                          
    }                                                                                                      
    catch (const vmime::exception& ex){std::cout << "Ex: " << ex.name() << "->" << ex.what() << std::endl;}
    catch (const std::exception& ex)  {std::cout << "Ex: " << ex.what() << std::endl;}                     
    catch (const std::string& exStr)  {std::cout << "Ex: " << exStr << std::endl;}                         
    catch (...)                       {std::cout << "Ex unknown" << std::endl;}                            
    return 1;                                                                                              
}

smtps:1->250-SIZE 35882577 smtps:1->250-8BITMIME smtps:1->250-AUTH LOGIN PLAIN XOAUTH2 PLAIN-CLIENTTOKEN XOAUTH smtps:1->250-ENHANCEDSTATUSCODES smtps:1->250-PIPELINING smtps:1->250-CHUNKING smtps:1->250 SMTPUTF8 smtps:1->(Client)AUTH PLAIN smtps:1->334 smtps:1->(Client){...SASL exchange: 40 bytes of data...} smtps:1->235 2.7.0 Accepted smtps:1->(Client)MAIL FROM:<vmimeö4711@gmail.com> SMTPUTF8 SIZE=333 smtps:1->(Client)RCPT TO:<vmimeö4711@gmail.com> smtps:1->(Client)DATA smtps:1->250 2.1.0 OK e2sm20508331wix.15 - gsmtp smtps:1->250 2.1.5 OK e2sm20508331wix.15 - gsmtp smtps:1->354 Go ahead e2sm20508331wix.15 - gsmtp smtps:1->(Client){...333 bytes of data...} smtps:1->(Client). smtps:1->250 2.0.0 OK 1430302709 e2sm20508331wix.15 - gsmtp smtps:1->(Client)QUIT smtps:1->(Client)Disconnecting

Now I built the same Message with the Message-Builder of vmime, and it is doing an explicit encoding of the to- and from-header-fields. I am not sure, If the encoding which vmime does is RFC-compliant, cause I can see here TWO encodings. In addition to that, my complete Hostname is added as domain-part to my E-Mail-Address? Is that correct?

#define VMIME_STATIC
#include <vmime/vmime.hpp>
#include <iostream>

struct MyCV : vmime::security::cert::defaultCertificateVerifier
{
    void verify(__attribute__((unused))vmime::shared_ptr<vmime::security::cert::certificateChain> c,
                __attribute__((unused))const vmime::string& h) {}
};

struct MyTF : vmime::net::tracerFactory
{
    struct MyT : vmime::net::tracer
    {
        MyT(const vmime::string& proto, const int cId) : m_p(proto), m_cId(cId) {}
        void traceSend(const vmime::string& l){traceReceive("(Client)" + l);}
        void traceReceive(const vmime::string& l){std::cout << m_p << ":" << m_cId << "->" << l << std::endl;}
        const vmime::string m_p;
        const int m_cId;
    };

    vmime::shared_ptr<vmime::net::tracer> create(vmime::shared_ptr<vmime::net::service> serv, const int cId)
    {
        return vmime::make_shared<MyT>(serv->getProtocolName(), cId);
    }
};

int main()
{
    try
    {
        vmime::shared_ptr<vmime::net::session> sess = vmime::make_shared<vmime::net::session>();
        sess->getProperties().setProperty("transport.protocol","smtps");
        vmime::shared_ptr<vmime::net::transport> trans = sess->getTransport();
        trans->setProperty("server.address","smtp.gmail.com");
        trans->setProperty("server.port",465);
        trans->setProperty("options.need-authentication", true);
        trans->setProperty("auth.username", "vmime4711@gmail.com");
        trans->setProperty("auth.password", "vmimetest");
        trans->setProperty("options.chunking",false);
        trans->setCertificateVerifier(vmime::make_shared<MyCV>());
        trans->setTracerFactory(vmime::make_shared<MyTF>());
        trans->connect();

        vmime::messageBuilder mb;

        vmime::text strAddressUTF8;
        strAddressUTF8.createFromString("vmimeö4711@gmail.com","utf-8");
        vmime::emailAddress mailAddress = vmime::emailAddress(strAddressUTF8.generate());

        vmime::text strNameUTF8;
        strNameUTF8.createFromString("vmime4711","utf-8");

        vmime::addressList addressListTo  = vmime::addressList();
        addressListTo.appendAddress(vmime::make_shared<vmime::mailbox>(strNameUTF8,mailAddress));

        mb.setExpeditor(vmime::mailbox(strNameUTF8,mailAddress));
        mb.setRecipients(addressListTo);

        vmime::text strSubjectUTF8;
        strSubjectUTF8.createFromString("Test","utf-8");
        mb.setSubject(strSubjectUTF8);

        mb.constructTextPart(vmime::mediaType(vmime::mediaTypes::TEXT, vmime::mediaTypes::TEXT_PLAIN));
        vmime::shared_ptr<vmime::textPart> textPart = vmime::dynamicCast<vmime::textPart>(mb.getTextPart());

        textPart->setCharset(vmime::charsets::UTF_8);
        textPart->setText(vmime::make_shared<vmime::stringContentHandler>("Bodycontent"));

        vmime::shared_ptr<vmime::message> msgFromBuilder = mb.construct();

        std::stringstream ssUTF8MailPlain;
        vmime::utility::outputStreamAdapter outRaw(ssUTF8MailPlain);
        msgFromBuilder->generate(outRaw);

        //Building msg-object again from the scratch and send it
        vmime::shared_ptr<vmime::message> msg = vmime::make_shared<vmime::message>();
        msg->parse(ssUTF8MailPlain.str());
        trans->send(msg);
        trans->disconnect();
        return 0;
    }
    catch (const vmime::exception& ex){std::cout << "Ex: " << ex.name() << "->" << ex.what() << std::endl;}
    catch (const std::exception& ex)  {std::cout << "Ex: " << ex.what() << std::endl;}
    catch (const std::string& exStr)  {std::cout << "Ex: " << exStr << std::endl;}
    catch (...)                       {std::cout << "Ex unknown" << std::endl;}
    return 1;
}

smtps:1->250-SIZE 35882577 smtps:1->250-8BITMIME smtps:1->250-AUTH LOGIN PLAIN XOAUTH2 PLAIN-CLIENTTOKEN XOAUTH smtps:1->250-ENHANCEDSTATUSCODES smtps:1->250-PIPELINING smtps:1->250-CHUNKING smtps:1->250 SMTPUTF8 smtps:1->(Client)AUTH PLAIN smtps:1->334 smtps:1->(Client){...SASL exchange: 40 bytes of data...} smtps:1->235 2.7.0 Accepted smtps:1->(Client)MAIL FROM:=?utf-8?Q?=3D=3Futf-8=3FQ=3Fvmime=3DC3=3DB64711=3D40gmail=3D2Ecom=3F=3D?=@ivrxxxx.a.office.xxxx.de SMTPUTF8 SIZE=481 smtps:1->(Client)RCPT TO:=?utf-8?Q?=3D=3Futf-8=3FQ=3Fvmime=3DC3=3DB64711=3D40gmail=3D2Ecom=3F=3D?=@ivrxxxx.a.office.xxxx.de smtps:1->(Client)DATA smtps:1->250 2.1.0 OK bm9sm28388801wjc.21 - gsmtp smtps:1->250 2.1.5 OK bm9sm28388801wjc.21 - gsmtp smtps:1->354 Go ahead bm9sm28388801wjc.21 - gsmtp smtps:1->(Client){...481 bytes of data...} smtps:1->(Client). smtps:1->250 2.0.0 OK 1430303653 bm9sm28388801wjc.21 - gsmtp smtps:1->(Client)QUIT smtps:1->(Client)Disconnecting

Are you able to verify, that this is the correct behavior? Maybe I am misusing the MessageBuilder, but If I look at the encoded string, it is encoded over TWO stages with the additional Hostname. (An exchange-server for example is returning now here an "invalid-address"-error!)

Stage1: =?utf-8?Q?=3D=3Futf-8=3FQ=3Fvmime=3DC3=3DB64711=3D40gmail=3D2Ecom=3F=3D?=@ivrxxxx.a.office.xxxx.de

Stage2: =?utf-8?Q?vmime=C3=B64711=40gmail=2Ecom?=@ivrxxxx.a.office.xxxx.de

Stage 3: <vmimeö4711@gmail.com@ivrxxxx.a.office.xxxx.de>

vincent-richard commented 9 years ago

Hi!

The first example is OK, as long as the SMTP server supports the SMTPUTF8 extension, which is the case. If not, then you can't use UTF8 in email addresses, and VMime can't do anything for it.

As for the second example, you should not call generate() on the "text" object for the email address:

    vmime::emailAddress mailAddress = vmime::emailAddress(strAddressUTF8.generate());

Instead, do this:

    vmime::emailAddress mailAddress(
            vmime::word("vmimeö4711", vmime::charset("utf-8")),
            vmime::word("gmail.com", vmime::charset("utf-8"))
    );

This way, there will be no additional encoding. I can't think of any real life usage of the generate() function on the user side of the library.

Vincent

SoulfreezerXP commented 9 years ago

Thank you so much for your fast support. After eliminating the generate-function() and inserting your code-fragment, the MessageBuilder is building the following mail:

Subject: Test From: "vmime4711" <=?utf-8?Q?vmime=C3=B64711?=@gmail.com> To: "vmime4711" <=?utf-8?Q?vmime=C3=B64711?=@gmail.com> Date: Thu, 30 Apr 2015 10:03:17 +0200 Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit

Bodycontent

This looks much better. The Headers are now correct encoded in qouted-printable, because an E-Mail should not contain any non-ASCII chars. Right?

But then, If I send this encoded mail via smtp, vmime extracts the from- and to-fields WITHOUT decoding it to real utf-8 and gives them qouted-printable-encoded to the smtp-server with SMTPUTF8-Parameter enabled? The MessageBuilder seems to build the mail correct, but the takeover of the from-/To-Fields to the smtp-server...is this correct/RFC-compliant? Or is all ok now?

smtps:1->(Client)MAIL FROM:<=?utf-8?Q?vmime=C3=B64711?=@gmail.com> SMTPUTF8 SIZE=353 smtps:1->(Client)RCPT TO:<=?utf-8?Q?vmime=C3=B64711?=@gmail.com> smtps:1->(Client)DATA smtps:1->250 2.1.0 OK n8sm1209469wiy.19 - gsmtp smtps:1->250 2.1.5 OK n8sm1209469wiy.19 - gsmtp smtps:1->354 Go ahead n8sm1209469wiy.19 - gsmtp smtps:1->(Client){...353 bytes of data...} smtps:1->(Client). smtps:1->250 2.0.0 OK 1430380998 n8sm1209469wiy.19 - gsmtp smtps:1->(Client)QUIT smtps:1->(Client)Disconnecting

vincent-richard commented 9 years ago

Hi!

The following is not valid:

smtps:1->(Client)MAIL FROM:<=?utf-8?Q?vmime=C3=B64711?=@gmail.com> SMTPUTF8 SIZE=353
smtps:1->(Client)RCPT TO:<=?utf-8?Q?vmime=C3=B64711?=@gmail.com>

When SMTPUTF8 is available, it should be:

smtps:1->(Client)MAIL FROM:<vmimeö4711@gmail.com> SMTPUTF8 SIZE=333
smtps:1->(Client)RCPT TO:<vmimeö4711@gmail.com>

What code did you use to send the message in the last case?

SoulfreezerXP commented 9 years ago

Hi Vincent,

I was using the example-code here from my initial issue-posting. Only modified mail-address.

#define VMIME_STATIC                                                                                        
#include <vmime/vmime.hpp>                                                                                  
#include <iostream>                                                                                         

struct MyCV : vmime::security::cert::defaultCertificateVerifier                                             
{                                                                                                           
    void verify(__attribute__((unused))vmime::shared_ptr<vmime::security::cert::certificateChain> c,        
                __attribute__((unused))const vmime::string& h) {}                                           
};                                                                                                          

struct MyTF : vmime::net::tracerFactory                                                                     
{                                                                                                           
    struct MyT : vmime::net::tracer                                                                         
    {                                                                                                       
        MyT(const vmime::string& proto, const int cId) : m_p(proto), m_cId(cId) {}                          
        void traceSend(const vmime::string& l){traceReceive("(Client)" + l);}                               
        void traceReceive(const vmime::string& l){std::cout << m_p << ":" << m_cId << "->" << l << std::endl
        const vmime::string m_p;                                                                            
        const int m_cId;                                                                                    
    };                                                                                                      

    vmime::shared_ptr<vmime::net::tracer> create(vmime::shared_ptr<vmime::net::service> serv, const int cId)
    {                                                                                                       
        return vmime::make_shared<MyT>(serv->getProtocolName(), cId);                                       
    }                                                                                                       
};                                                                                                          

int main()                                                                                                  
{                                                                                                           
    try                                                                                                     
    {                                                                                                       
        vmime::shared_ptr<vmime::net::session> sess = vmime::make_shared<vmime::net::session>();            
        sess->getProperties().setProperty("transport.protocol","smtps");                                    
        vmime::shared_ptr<vmime::net::transport> trans = sess->getTransport();                              
        trans->setProperty("server.address","smtp.gmail.com");                                              
        trans->setProperty("server.port",465);                                                              
        trans->setProperty("options.need-authentication", true);                                            
        trans->setProperty("auth.username", "vmime4711@gmail.com");                                         
        trans->setProperty("auth.password", "vmimetest");                                                   
        trans->setProperty("options.chunking",false);                                                       
        trans->setCertificateVerifier(vmime::make_shared<MyCV>());                                          
        trans->setTracerFactory(vmime::make_shared<MyTF>());                                                
        trans->connect();                                                                                   

        vmime::messageBuilder mb;                                                                           

        //vmime::text strAddressUTF8;                                                                       
        //strAddressUTF8.createFromString("vmimeö4711@gmail.com","utf-8");                                 
        //vmime::emailAddress mailAddress = vmime::emailAddress(strAddressUTF8.generate());                 

    //REPLACED PART HERE                                                                                    
    vmime::emailAddress mailAddress(                                                                        
            vmime::word("vmimeö4711", vmime::charset("utf-8")),                                            
            vmime::word("gmail.com", vmime::charset("utf-8"))                                               
        );                                                                                                  

        vmime::text strNameUTF8;                                                                            
        strNameUTF8.createFromString("vmime4711","utf-8");                                                  

        vmime::addressList addressListTo  = vmime::addressList();                                           
        addressListTo.appendAddress(vmime::make_shared<vmime::mailbox>(strNameUTF8,mailAddress));           

        mb.setExpeditor(vmime::mailbox(strNameUTF8,mailAddress));                                           
        mb.setRecipients(addressListTo);                                                                    

        vmime::text strSubjectUTF8;                                                                         
        strSubjectUTF8.createFromString("Test","utf-8");                                                    
        mb.setSubject(strSubjectUTF8);                                                                      

        mb.constructTextPart(vmime::mediaType(vmime::mediaTypes::TEXT, vmime::mediaTypes::TEXT_PLAIN));     
        vmime::shared_ptr<vmime::textPart> textPart = vmime::dynamicCast<vmime::textPart>(mb.getTextPart());

        textPart->setCharset(vmime::charsets::UTF_8);                                                       
        textPart->setText(vmime::make_shared<vmime::stringContentHandler>("Bodycontent"));                  

        vmime::shared_ptr<vmime::message> msgFromBuilder = mb.construct();                                  

        std::stringstream ssUTF8MailPlain;                                                                  
        vmime::utility::outputStreamAdapter outRaw(ssUTF8MailPlain);                                        
        msgFromBuilder->generate(outRaw);                                                                   

    std::cout << ssUTF8MailPlain.str() << std::endl; //PRINTED GENERATED MAIL HERE                          

        //Building msg-object again from the scratch and send it                                            
        vmime::shared_ptr<vmime::message> msg = vmime::make_shared<vmime::message>();                       
        msg->parse(ssUTF8MailPlain.str());                                                                  
        trans->send(msg);                                                                                   
        trans->disconnect();                                                                                
        return 0;                                                                                           
    }                                                                                                       
    catch (const vmime::exception& ex){std::cout << "Ex: " << ex.name() << "->" << ex.what() << std::endl;} 
    catch (const std::exception& ex)  {std::cout << "Ex: " << ex.what() << std::endl;}                      
    catch (const std::string& exStr)  {std::cout << "Ex: " << exStr << std::endl;}                          
    catch (...)                       {std::cout << "Ex unknown" << std::endl;}                             
    return 1;                                                                                               
}
vincent-richard commented 9 years ago

Hi!

There was a bug in email address parsing: RFC-2047-encoded local parts were not supported when internationalized email support was not enabled.

The following commit should fix this: https://github.com/kisli/vmime/commit/e88b8eeac27ec3a829a4622c861a9018b9baa329

Vincent

SoulfreezerXP commented 9 years ago

Thank you so much for this bugfix. I will test it the next days!

SoulfreezerXP commented 9 years ago

Now, recompiled and linked the last Code-Example with the newest Version (Master) from Git, the Bug disappears! Thanks a lot for this fast support!

Greetz Soul

smtps:1->250-SIZE 35882577 smtps:1->250-8BITMIME smtps:1->250-AUTH LOGIN PLAIN XOAUTH2 PLAIN-CLIENTTOKEN XOAUTH smtps:1->250-ENHANCEDSTATUSCODES smtps:1->250-PIPELINING smtps:1->250-CHUNKING smtps:1->250 SMTPUTF8 smtps:1->(Client)AUTH PLAIN smtps:1->334 smtps:1->(Client){...SASL exchange: 40 bytes of data...} smtps:1->235 2.7.0 Accepted smtps:1->(Client)MAIL FROM:<vmimeö4711@gmail.com> SMTPUTF8 SIZE=319 smtps:1->(Client)RCPT TO:<vmimeö4711@gmail.com> smtps:1->(Client)DATA smtps:1->250 2.1.0 OK yr1sm20598115wjc.37 - gsmtp smtps:1->250 2.1.5 OK yr1sm20598115wjc.37 - gsmtp smtps:1->354 Go ahead yr1sm20598115wjc.37 - gsmtp smtps:1->(Client){...319 bytes of data...} smtps:1->(Client). smtps:1->250 2.0.0 OK 1430745497 yr1sm20598115wjc.37 - gsmtp smtps:1->(Client)QUIT smtps:1->(Client)Disconnecting

vincent-richard commented 9 years ago

Fine! Thanks for testing.