e-mail messages with application/rtf body are imported as attachments, not message body

vsessink commented 2 months ago

While importing an e-mail-archive in the (IMHO cursed) .PST-format, I came across a mailbox having all application/rtf for body type.

Content-Type: application/rtf
Content-Transfer-Encoding: base64
Content-Disposition: attachment; 
        filename*=utf-8''rtf-body.rtf;
        filename="rtf-body.rtf"

Yep, that's right: Content-Disposition: attachment, but still this is the actual e-mail body.

Now in Aleph, these messages will show up as empty, with rtf-body.rtf document as attachment.

I tried to work around it by unpacking the mail archive manually with readpst; then fixing the messages with a small python script (essentially replacing the rtf part with an html part. I used python's email.parser and simply checked if the first content_type would be application/rtf - if so, pipe that through unrtf and repack the message. Filthy, but working for the mail box itself).

This workaround would not help in Aleph, because the mime detection wizardry afterwards recognized text/html for mime type, instead of message/rfc822 - and actual attachments of the message would not be recognized anymore.

The latter may count as a separate bug: a message that starts with the following should IMHO not be detected as text/html?

Status: RO
User-Agent: none
From: "Firstname Lastname" <MAILER-DAEMON>
Subject: FW: Ticket 08-05
To:  Name (Company Name)
Date: Tue, 09 May 2022 14:41:50 +0000
Message-Id: <AM5PR04MB53161D02E214FEEC35C2156FB6466@AM0PR04MB9122.eurprd02.prod.outlook.com>
X-libpst-forensic-sender: /O=EXCHANGELABS/OU=EXCHANGE ADMINISTRATIVE GROUP (FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=09C4BB2213F35544FEBBBBF1FD14B522
MIME-Version: 1.0
Content-Type: multipart/mixed;
        boundary="--boundary-LibPST-iamunique-887075155_-_-"

----boundary-LibPST-iamunique-887075155_-_-
Content-Type: text/html; charset="utf-8"

<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}

vsessink commented 2 months ago

This (previously WIP, now abandoned) mentions the same problems https://github.com/alephdata/ingest-file/pull/20

Where is the mime detection done? I think it could work to try fix the output of readpst - by adding transport headers or otherwise; fix the RTF-parts of the e-mails, too. As I already walk over all e-mails to fix the RTF-parts, adding required headers for mime detection (message/rfc822 instead of text/html) could work, too.

vsessink commented 2 months ago

Messages can pretty easily be "tricked" into being message/rfc822, by simply adding Received: from localhost (127.0.0.1) at the top of the message. IMHO as a workaround for the current state of things, this could be done right after readpst. I will investigate.

vsessink commented 2 months ago

In order to fix messages that have an RTF-only message body, I'm manually starting a Python script:

#!/usr/bin/python3
import base64
import os
import sys
import re
import mimetypes
import email
from email.policy import default
from email.parser import BytesParser
import subprocess

plcy=default.clone(refold_source='none')
for fname in sys.argv[1:]:
  try:
    mail=open(fname,'rb')
  except:
    print(fname, "not found.")
    continue
  msg = BytesParser(policy=plcy).parse(mail)
  mail.close()
  totaal=list(msg.walk())
  if (len(totaal)<2):
    continue
  if (totaal[1].get_content_type() == 'application/rtf'):
    print("Converting", fname)
    html=subprocess.run(['/usr/bin/unrtf'], input=totaal[1].get_content(), capture_output=True).stdout
    totaal[1].set_content(html, maintype='text',subtype='html')
    try:
      mail=open(fname,'w')
    except:
      print("Error writing")
      continue
    print(totaal[0], file=mail)
    mail.close()

It's a hack. But it works and it really helps the search process. This could be run right after readpst but I really don't think this is production quality. Anyway, maybe it helps someone make a proper fix.

vsessink commented 2 weeks ago

OK, here's more analysis and an awful corner case. I'm documenting it here because I don't think there's a better place. I unpacked a pst file with the regular readpst -e -D -8 -cv. One of these messages contains two message/rfc822 attachments having rtf-body.rtf for content type. So the script above should be made recursive.

Then the awful part of the finding is, that my attachments begin with

Content-Type: message/rfc822

>From "mailaddress@example.com" Tue Oct  4 14:22:48 2023

That shouldn't happen, the readpst man page says that for -e This format has no from quoting. (Where from quoting means prepending the word From with a > character). However, it apparently does. You must remove the >, otherwise the EmailMessage is wrongly interpreted.

vsessink commented 2 weeks ago

Looking briefly, the From quoting problem is in readpst, libpst/src/readpst.c, where write_embedded_message writes out messages with the "From" quoting parameter ("embedding") being 1.

write_normal_email(f_output, "", item, MODE_NORMAL, 0, pf, save_rtf,
 1, extra_mime_headers);

alephdata / ingest-file

e-mail messages with application/rtf body are imported as attachments, not message body #618