search

jakartaee / mail-api

Jakarta Mail Specification project
https://jakartaee.github.io/mail-api
Other
244 stars 101 forks source link

SharedByteArrayInputStream instead mutipart with mail.pop3.forgettopheaders=true #613

Open demarco88 opened 2 years ago

demarco88 commented 2 years ago

Hi. We face the problem, that we ant to use the flag mail.pop3.forgettopheaders=true in order to force new header request in RETR command, because of header changes of some providers between the TOP and the RETR Command.

At TOP Command the content-type is: Content-Type: application/pkcs7-mime; smime-type=authenticated-enveloped-data; name=smime.p7m At RETR Command the content-type is: Content-Type: multipart/mixed; boundary="----=_Part_45_761695249.1657878900597" Because between TOP Command and RETR Command the provider packed the mail inside a new mail envelope. This behaviour is part of a german healthcare communication standard specification called KIM.

Now when parsing the content of the email after the RETR Command content-type of the message is application/pkcs7-mime but it should be multipart/mixed. Further the getContent() delivers a SharedByteArrayInputStream instead of a Multipart-Object. Seams to be a bug in combination with forgettopheaders.

If we switch off the forgettopheaders parsing the content is working, but we have the "wrong" (first received ) header informations. If we switch of the mail.pop3.disablecapa or activate the mail.pop3.disabletop then we force the api to use directly the RETR Coammnd. In this case the email is also parsed correctly. But it isn't a good solution for us because we need to filter out some emails in case of specific header informations.

Tested with version 1.6.4 and also 1.6.7.

regards Marco

jbescos commented 2 years ago

Hi @demarco88 are you able to provide a reproducer or some example code so I can get better understanding?.

demarco88 commented 2 years ago

Hi @jbescos. Thanks for your request for additional information. It seems to be be a common problem of the cached headers if the RETR command delivers complete other headers than the TOP command. Especially when the content-type of the first part of the message has changed. And forgettopheaders=true does not help to fix the problem. Following simple parts of an example code. 

public void receiveemails(Account account) throws Exception
{
  this.pop3properties.put("mail.pop3.ssl.socketFactory", this.sslsocketfactory);
  this.pop3properties.put("mail.pop3.disablecapa", true);
  this.pop3properties.put("mail.pop3.disabletop", false);
  this.pop3properties.put("mail.pop3.forgettopheaders", false);
  this.pop3properties.put("mail.debug", "true");

  Session session = getSession(this.pop3properties, account.getUser(), account.getPassword());
  URLName url = new URLName("pop3", pop3properties.getProperty("mail.pop3.host"),
      Integer.parseInt(pop3properties.getProperty("mail.pop3.port")), "", account.getUser(), account.getPassword());
  POP3Store store = (POP3Store)session.getStore(url);
  store.connect();
  POP3Folder folder = (POP3Folder)store.getFolder("INBOX");
  folder.open(Folder.READ_WRITE);

  Message[] msgs = folder.getMessages();
  for(int a = 0;a < msgs.length;a++)
    traverse(msgs[a]);
  folder.close(false);
}

private static void traverse(Part p) throws Exception
{
  String contenttype = p.getContentType();
  Object o = p.getContent();
  System.out.println("MESSAGE: content-type:" + contenttype + " object class:" + o.getClass());
  if(o instanceof Multipart)
  {
    Multipart mp = (Multipart)o;
    int count = mp.getCount();
    // recursive Aufruf
    for(int i = 0;i < count;i++)
      traverse(mp.getBodyPart(i));
  }

  if(o instanceof MimeMessage)
  {
    MimeMessage att = (MimeMessage)o;
    traverse(att);
  }
  return;
}

Furthermore the debug protocol of this execution. Some data are anonymized. The complete base-64 content is cutted (MIAGCyqGS.....................CUTTED...==). It does'nt make sence to post it here completely.

DEBUG: Jakarta Mail version 1.6.4
DEBUG: successfully loaded resource: /META-INF/javamail.default.providers
DEBUG: Tables of loaded providers
DEBUG: Providers Listed By Class Name: {com.sun.mail.smtp.SMTPTransport=javax.mail.Provider[TRANSPORT,smtp,com.sun.mail.smtp.SMTPTransport,Oracle], com.sun.mail.imap.IMAPSSLStore=javax.mail.Provider[STORE,imaps,com.sun.mail.imap.IMAPSSLStore,Oracle], com.sun.mail.pop3.POP3Store=javax.mail.Provider[STORE,pop3,com.sun.mail.pop3.POP3Store,Oracle], com.sun.mail.smtp.SMTPSSLTransport=javax.mail.Provider[TRANSPORT,smtps,com.sun.mail.smtp.SMTPSSLTransport,Oracle], com.sun.mail.imap.IMAPStore=javax.mail.Provider[STORE,imap,com.sun.mail.imap.IMAPStore,Oracle], com.sun.mail.pop3.POP3SSLStore=javax.mail.Provider[STORE,pop3s,com.sun.mail.pop3.POP3SSLStore,Oracle]}
DEBUG: Providers Listed By Protocol: {imap=javax.mail.Provider[STORE,imap,com.sun.mail.imap.IMAPStore,Oracle], smtp=javax.mail.Provider[TRANSPORT,smtp,com.sun.mail.smtp.SMTPTransport,Oracle], pop3=javax.mail.Provider[STORE,pop3,com.sun.mail.pop3.POP3Store,Oracle], imaps=javax.mail.Provider[STORE,imaps,com.sun.mail.imap.IMAPSSLStore,Oracle], smtps=javax.mail.Provider[TRANSPORT,smtps,com.sun.mail.smtp.SMTPSSLTransport,Oracle], pop3s=javax.mail.Provider[STORE,pop3s,com.sun.mail.pop3.POP3SSLStore,Oracle]}
DEBUG: successfully loaded resource: /META-INF/javamail.default.address.map
DEBUG: getProvider() returning javax.mail.Provider[STORE,pop3,com.sun.mail.pop3.POP3Store,Oracle]
DEBUG POP3: mail.pop3.rsetbeforequit: false
DEBUG POP3: mail.pop3.disabletop: false
DEBUG POP3: mail.pop3.forgettopheaders: false
DEBUG POP3: mail.pop3.cachewriteto: false
DEBUG POP3: mail.pop3.filecache.enable: false
DEBUG POP3: mail.pop3.keepmessagecontent: false
DEBUG POP3: mail.pop3.starttls.enable: false
DEBUG POP3: mail.pop3.starttls.required: false
DEBUG POP3: mail.pop3.finalizecleanclose: false
DEBUG POP3: mail.pop3.apop.enable: false
DEBUG POP3: mail.pop3.disablecapa: true
DEBUG POP3: connecting to host "testserver.fancy.firm.net", port 995, isSSL true
+OK KOM-LE Clientmodul POP3
DEBUG POP3: authentication command trace suppressed
DEBUG POP3: authentication command succeeded
STAT
+OK 1 7424
NOOP
+OK
TOP 1 0
+OK
Return-Path: <test-email@anonym.kim.telematik-test>
Delivered-To: test-email@anonym.kim.telematik-test
Received: from gtegklve09183otherfancy.server.somewhere.de ([10.30.8.26])
    (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
    by gtegklvd09182otherfancy.server.somewhere.de with LMTPS
    id JG60JYd83mIprQEAz3/SOw
    (envelope-from <test-email@anonym.kim.telematik-test>)
    for <test-email@anonym.kim.telematik-test>; Mon, 25 Jul 2022 13:20:39 +0200
Received: from mail.arv.kim.telematik-test ([10.30.8.26])
    (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
    by gtegklve09183otherfancy.server.somewhere.de with LMTPS
    id 3LyGI4d83mLb2gEAU3MrGQ
    (envelope-from <test-email@anonym.kim.telematik-test>)
    for <test-email@anonym.kim.telematik-test>; Mon, 25 Jul 2022 13:20:39 +0200
Received: from unknown (unknown [10.23.16.42])
    by mail.arv.kim.telematik-test (Postfix) with SMTP id 075E8C1
    for <test-email@anonym.kim.telematik-test>; Mon, 25 Jul 2022 13:20:12 +0200 (CEST)
Date: Mon, 25 Jul 2022 13:20:01 +0200 (CEST)
From: test-email@anonym.kim.telematik-test
To: test-email@anonym.kim.telematik-test
Message-Id: <2070160397.1.1658748010950@W2025212.fancy.firm.net>
Subject: KOM-LE-Nachricht
MIME-Version: 1.0
Content-Type: application/pkcs7-mime; name=smime.p7m; 
    smime-type=authenticated-enveloped-data
Content-Transfer-Encoding: base64
X-KOM-LE-Version: 1.0
Content-Disposition: attachment; filename=smime.p7m
X-KIM-Dienstkennung: Selbsttest;Lieferung;V1.0

.
RETR 1
+OK
Date: Mon, 25 Jul 2022 13:20:01 +0200 (CEST)
From: test-email@anonym.kim.telematik-test
To: test-email@anonym.kim.telematik-test
Subject: Die Nachricht konnte nicht entschlüsselt werden.
MIME-Version: 1.0
Content-Type: multipart/mixed; 
    boundary="----=_Part_286_2060398125.1658750487374"

------=_Part_286_2060398125.1658750487374
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Der f=C3=BCr die Entschl=C3=BCsselung der Nachricht ben=C3=B6tigte Schl=C3=
=BCssel wurde nicht gefunden. =C3=9Cberpr=C3=BCfen Sie ob die entsprechende=
 Karte gesteckt ist und leiten Sie diese Nachricht an Ihre eigene E-Mail-Ad=
resse (test-email@anonym.kim.telematik-test) weiter. Beim n=C3=A4chsten Abho=
len wird der Entschl=C3=BCsselungsvorgang wiederholt.
------=_Part_286_2060398125.1658750487374
Content-Type: message/rfc822

Return-Path: <test-email@anonym.kim.telematik-test>
Delivered-To: test-email@anonym.kim.telematik-test
Received: from gtegklve09183otherfancy.server.somewhere.de ([10.30.8.26])
    (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
    by gtegklvd09182otherfancy.server.somewhere.de with LMTPS
    id JG60JYd83mIprQEAz3/SOw
    (envelope-from <test-email@anonym.kim.telematik-test>)
    for <test-email@anonym.kim.telematik-test>; Mon, 25 Jul 2022 13:20:39 +0200
Received: from mail.arv.kim.telematik-test ([10.30.8.26])
    (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
    by gtegklve09183otherfancy.server.somewhere.de with LMTPS
    id 3LyGI4d83mLb2gEAU3MrGQ
    (envelope-from <test-email@anonym.kim.telematik-test>)
    for <test-email@anonym.kim.telematik-test>; Mon, 25 Jul 2022 13:20:39 +0200
Received: from unknown (unknown [10.23.16.42])
    by mail.arv.kim.telematik-test (Postfix) with SMTP id 075E8C1
    for <test-email@anonym.kim.telematik-test>; Mon, 25 Jul 2022 13:20:12 +0200 (CEST)
Date: Mon, 25 Jul 2022 13:20:01 +0200 (CEST)
From: test-email@anonym.kim.telematik-test
To: test-email@anonym.kim.telematik-test
Message-Id: <2070160397.1.1658748010950@W2025212.fancy.firm.net>
Subject: KOM-LE-Nachricht
MIME-Version: 1.0
Content-Type: application/pkcs7-mime; name=smime.p7m; 
    smime-type=authenticated-enveloped-data
Content-Transfer-Encoding: base64
X-KOM-LE-Version: 1.0
Content-Disposition: attachment; filename=smime.p7m
X-KIM-Dienstkennung: Selbsttest;Lieferung;V1.0

MIAGCyqGS.....................CUTTED...==

------=_Part_286_2060398125.1658750487374--

.
MESSAGE: content-type:application/pkcs7-mime; name=smime.p7m; 
    smime-type=authenticated-enveloped-data object class:class com.sun.mail.util.BASE64DecoderStream
QUIT
+OK Logging out.
2

In the RETR Command we first have a multipart/mixed. Including a part text/plain and message/rfc822. But the additional debug logging in the travers method said:

MESSAGE: content-type:application/pkcs7-mime; name=smime.p7m; 
    smime-type=authenticated-enveloped-data object class:class com.sun.mail.util.BASE64DecoderStream

if this example is executed with the mail.pop3.forgettopheaders=true we will get the same output except the Type of the content object:

MESSAGE: content-type:application/pkcs7-mime; name=smime.p7m; 
    smime-type=authenticated-enveloped-data object class:class javax.mail.util.SharedByteArrayInputStream

Now the working example: Only think that seems to work is setting the mail.pop3.disabletop to true. But that's no good option for us. Then it will produce the following output:

DEBUG: Jakarta Mail version 1.6.4
DEBUG: successfully loaded resource: /META-INF/javamail.default.providers
DEBUG: Tables of loaded providers
DEBUG: Providers Listed By Class Name: {com.sun.mail.smtp.SMTPTransport=javax.mail.Provider[TRANSPORT,smtp,com.sun.mail.smtp.SMTPTransport,Oracle], com.sun.mail.imap.IMAPSSLStore=javax.mail.Provider[STORE,imaps,com.sun.mail.imap.IMAPSSLStore,Oracle], com.sun.mail.pop3.POP3Store=javax.mail.Provider[STORE,pop3,com.sun.mail.pop3.POP3Store,Oracle], com.sun.mail.smtp.SMTPSSLTransport=javax.mail.Provider[TRANSPORT,smtps,com.sun.mail.smtp.SMTPSSLTransport,Oracle], com.sun.mail.imap.IMAPStore=javax.mail.Provider[STORE,imap,com.sun.mail.imap.IMAPStore,Oracle], com.sun.mail.pop3.POP3SSLStore=javax.mail.Provider[STORE,pop3s,com.sun.mail.pop3.POP3SSLStore,Oracle]}
DEBUG: Providers Listed By Protocol: {imap=javax.mail.Provider[STORE,imap,com.sun.mail.imap.IMAPStore,Oracle], smtp=javax.mail.Provider[TRANSPORT,smtp,com.sun.mail.smtp.SMTPTransport,Oracle], pop3=javax.mail.Provider[STORE,pop3,com.sun.mail.pop3.POP3Store,Oracle], imaps=javax.mail.Provider[STORE,imaps,com.sun.mail.imap.IMAPSSLStore,Oracle], smtps=javax.mail.Provider[TRANSPORT,smtps,com.sun.mail.smtp.SMTPSSLTransport,Oracle], pop3s=javax.mail.Provider[STORE,pop3s,com.sun.mail.pop3.POP3SSLStore,Oracle]}
DEBUG: successfully loaded resource: /META-INF/javamail.default.address.map
DEBUG: getProvider() returning javax.mail.Provider[STORE,pop3,com.sun.mail.pop3.POP3Store,Oracle]
DEBUG POP3: mail.pop3.rsetbeforequit: false
DEBUG POP3: mail.pop3.disabletop: true
DEBUG POP3: mail.pop3.forgettopheaders: false
DEBUG POP3: mail.pop3.cachewriteto: false
DEBUG POP3: mail.pop3.filecache.enable: false
DEBUG POP3: mail.pop3.keepmessagecontent: false
DEBUG POP3: mail.pop3.starttls.enable: false
DEBUG POP3: mail.pop3.starttls.required: false
DEBUG POP3: mail.pop3.finalizecleanclose: false
DEBUG POP3: mail.pop3.apop.enable: false
DEBUG POP3: mail.pop3.disablecapa: true
DEBUG POP3: connecting to host "testserver.fancy.firm.net", port 995, isSSL true
+OK KOM-LE Clientmodul POP3
DEBUG POP3: authentication command trace suppressed
DEBUG POP3: authentication command succeeded
STAT
+OK 1 7424
NOOP
+OK
RETR 1
+OK
Date: Mon, 25 Jul 2022 13:20:01 +0200 (CEST)
From: test-email@anonym.kim.telematik-test
To: test-email@anonym.kim.telematik-test
Subject: Die Nachricht konnte nicht entschlüsselt werden.
MIME-Version: 1.0
Content-Type: multipart/mixed; 
    boundary="----=_Part_293_810607501.1658751499303"

------=_Part_293_810607501.1658751499303
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Der f=C3=BCr die Entschl=C3=BCsselung der Nachricht ben=C3=B6tigte Schl=C3=
=BCssel wurde nicht gefunden. =C3=9Cberpr=C3=BCfen Sie ob die entsprechende=
 Karte gesteckt ist und leiten Sie diese Nachricht an Ihre eigene E-Mail-Ad=
resse (test-email@anonym.kim.telematik-test) weiter. Beim n=C3=A4chsten Abho=
len wird der Entschl=C3=BCsselungsvorgang wiederholt.
------=_Part_293_810607501.1658751499303
Content-Type: message/rfc822

Return-Path: <test-email@anonym.kim.telematik-test>
Delivered-To: test-email@anonym.kim.telematik-test
Received: from gtegklve09183otherfancy.server.somewhere.de ([10.30.8.26])
    (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
    by gtegklvd09182otherfancy.server.somewhere.de with LMTPS
    id JG60JYd83mIprQEAz3/SOw
    (envelope-from <test-email@anonym.kim.telematik-test>)
    for <test-email@anonym.kim.telematik-test>; Mon, 25 Jul 2022 13:20:39 +0200
Received: from mail.arv.kim.telematik-test ([10.30.8.26])
    (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
    by gtegklve09183otherfancy.server.somewhere.de with LMTPS
    id 3LyGI4d83mLb2gEAU3MrGQ
    (envelope-from <test-email@anonym.kim.telematik-test>)
    for <test-email@anonym.kim.telematik-test>; Mon, 25 Jul 2022 13:20:39 +0200
Received: from unknown (unknown [10.23.16.42])
    by mail.arv.kim.telematik-test (Postfix) with SMTP id 075E8C1
    for <test-email@anonym.kim.telematik-test>; Mon, 25 Jul 2022 13:20:12 +0200 (CEST)
Date: Mon, 25 Jul 2022 13:20:01 +0200 (CEST)
From: test-email@anonym.kim.telematik-test
To: test-email@anonym.kim.telematik-test
Message-Id: <2070160397.1.1658748010950@W2025212.fancy.firm.net>
Subject: KOM-LE-Nachricht
MIME-Version: 1.0
Content-Type: application/pkcs7-mime; name=smime.p7m; 
    smime-type=authenticated-enveloped-data
Content-Transfer-Encoding: base64
X-KOM-LE-Version: 1.0
Content-Disposition: attachment; filename=smime.p7m
X-KIM-Dienstkennung: Selbsttest;Lieferung;V1.0

MIAGCyqGS.....................CUTTED...==

------=_Part_293_810607501.1658751499303--

.
MESSAGE: content-type:multipart/mixed; 
    boundary="----=_Part_293_810607501.1658751499303" object class:class javax.mail.internet.MimeMultipart
MESSAGE: content-type:text/plain; charset=utf-8 object class:class java.lang.String
MESSAGE: content-type:message/rfc822 object class:class javax.mail.internet.MimeMessage
MESSAGE: content-type:application/pkcs7-mime; name=smime.p7m; 
    smime-type=authenticated-enveloped-data object class:class com.sun.mail.util.BASE64DecoderStream
QUIT
+OK Logging out.
2

Now the message has the correct parts. mail.pop3.forgettopheaders does not help or it's a bug in there.

MESSAGE: content-type:multipart/mixed; 
    boundary="----=_Part_293_810607501.1658751499303" object class:class javax.mail.internet.MimeMultipart
MESSAGE: content-type:text/plain; charset=utf-8 object class:class java.lang.String
MESSAGE: content-type:message/rfc822 object class:class javax.mail.internet.MimeMessage
MESSAGE: content-type:application/pkcs7-mime; name=smime.p7m; 
    smime-type=authenticated-enveloped-data object class:class com.sun.mail.util.BASE64DecoderStream
Codein1 commented 1 year ago

@demarco88 Just ran in the same issue. How did you solve it?

jbescos commented 6 months ago

I am debugging this. mail.pop3.forgettopheaders=true is ignored when mail.pop3.disabletop is not true.

This flag has to be true, and that can only happen if mail.pop3.disabletop=true or there is no TOP message. https://github.com/eclipse-ee4j/angus-mail/blob/62aae89956dfa26c92e854c04e2ac9c14528a10d/providers/pop3/src/main/java/org/eclipse/angus/mail/pop3/POP3Message.java#L641

You said that mail.pop3.disabletop=true is not a good option for you, but I don't see why is not an option. Could you help me to understand that better, please?.