[Inbound] Apple Mail inline attachments not taken into account

[martinezleoml]

Hi there!

We've figured out that inbound email processing of emails sent through Apple Mail app (tested on macOS) resulted on inline attachments (like images) not being taken into account at all by django-anymail.

• Anymail version: 10.0
• ESP: AWS SES
• Django version: 4.1.3
• macOS version: Ventura 13.4.1
• Apple Mail version: 16.0

Reproduction

1. Send a basic email with picture(s) using Apple Mail:

2. Copy-paste the resulting raw message content (sent by Apple Mail) into a file named message.raw:

   
   Return-Path: <redacted@example.com>
   Received: from smtpclient.apple ([REDACTED])
       by smtp.gmail.com with ESMTPSA id h9-REDACTED.88.2023.07.20.08.16.22
       for <redacted@example.com>
       (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
       Thu, 20 Jul 2023 08:16:22 -0700 (PDT)
   From: "REDACTED" <redacted@example.com>
   Content-Type: multipart/mixed; boundary="Apple-Mail=_B8C68BCC-061A-452F-A5CD-474B952DC8B5"
   Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.600.7\))
   Subject: Apple Mail attachment
   Message-Id: <FEFBB6BE-E215-403D-8FC8-38A51F0F07F3@example.com>
   Date: Thu, 20 Jul 2023 17:16:12 +0200
   To: "REDACTED" <redacted@example.com>
   X-Mailer: Apple Mail (2.3731.600.7)

--Apple-Mail=_B8C68BCC-061A-452F-A5CD-474B952DC8B5 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii

Some text before rectangle

--Apple-Mail=_B8C68BCC-061A-452F-A5CD-474B952DC8B5 Content-Disposition: inline; filename=rectangle.png Content-Type: image/png; x-unix-mode=0644; name="rectangle.png" Content-Transfer-Encoding: base64

iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAACXBIWXMAAAsTAAALEwEAmpwYAAAA AXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAQSURBVHgBAQUA+v8AAAAA/wEEAQB5fl4xAAAA AElFTkSuQmCC

--Apple-Mail=_B8C68BCC-061A-452F-A5CD-474B952DC8B5 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii

Some text after rectangle --Apple-Mail=_B8C68BCC-061A-452F-A5CD-474B952DC8B5--

3. Run the following Python script:
```python
from anymail.inbound import AnymailInboundMessage

with open("message.raw", "r") as fp:
  message: AnymailInboundMessage = AnymailInboundMessage.parse_raw_mime_file(fp)

  print(f"Attachments count: {len(message.attachments)}")
  print(f"Inline attachments count: {len(message.inline_attachments)}")
  print("")

  for part in message.walk():
      print(f"Part - {part['Content-Disposition']}")

The output shows that no attachment or inline attachment has been detected, even if we can read the attachment walking through message parts:

Attachments count: 0
Inline attachments count: 0

Part - None
Part - None
Part - inline; filename="rectangle.png"
Part - None

Resolution

I think that "inline attachments" without a Content-ID header should be considered as regular attachments, as Gmail do when receiving this kind of email:

[medmunds]

Thanks for investigating, and for including a full raw email demonstrating the problem.

My initial reaction was, Apple Mail is doing something weird, so yeah, whatever workaround makes sense. But the more I investigated, Apple Mail's use of multipart/mixed to interleave text/plain and inline image parts is a valid approach. The inline images don't need content-ids, because they're not being referenced elsewhere, and multipart/mixed is defined to present its parts in sequential order. (See RFC 2046 section 5.1.3—and W3C's older version that clarifies multipart/mixed children are "intended to be displayed serially." Also RFC 2183 sections 2.1-2.2 which specify content-disposition.)

Arguably, Gmail is doing the wrong thing by presenting the inline images at the end of the message as attachments.

I think the bug here is that Anymail's inline_attachments doesn't include these inline images that don't have content-ids, even though they are inline content. (And in fact, it can't, because inline_attachments is actually a dict mapping content-id to MIMEPart.)

The underlying problem is that Anymail's inline_attachments dict doesn't parallel its attachments list, leading to understandable confusion. (Something similar came up in #229.) Also "inline attachment" isn't really a thing—a MIMEPart is either inline or an attachment (or neither), never both.

Proposal:

• Rename Anymail's inline_attachments to content_id_map (or cid_map?), reflecting what it actually implements. (Keep inline_attachments as a deprecated alias for content_id_map, issuing a warning on use.)
• Rename Anymail's is_inline_attachment() to is_inline(). (And keep the original as a deprecated alias.)
• Add a new inlines list property that returns a sequence of parts that are inline (paralleling the behavior of attachments).
• Derive AnymailInboundMessage from Python's email.message.EmailMessage, rather than email.message.Message. (Current implementation is a holdover from Python 2.7 support.) Since EmailMessage already implements is_attachment() (via MIMEPart), get rid of the AnymailInboundMessage version.

[martinezleoml]

Thanks for the follow up @medmunds.

I was not sure what was the intended behaviour between displaying images inline or consider it as raw attachments, so thanks for clarifying this. 🙂

Your proposal works for me! Would you want some help to refactor?

[medmunds]

Would you want some help to refactor?

Yes, please! If you want to open a PR with any or all of the above, that would be much appreciated.

[martinezleoml]

@medmunds Here is a start #330 ;)

[medmunds]

Resolved by #330.