Better handling of multi_body_part Email observables (currently not conforming to STIX2.1)

[screencoffee]

Description

After playing around with parsing Emails and turning them into stix bundles to be uploaded to OpenCTI, I have come to the conclusion that the current implementation does not conform to the current STIX2.1 standard.

This is due to the "Email MIME Component Type" is a sort of sub-container inside the "Email Message Object". The "Email MIME Component Type" doesn't actually have an id like other observables.

Relevant links: STIX2.1 Specification of Email Message Object STIX2.1 Specification of Email MIME Component Type

Environment

1. OS (where OpenCTI server runs): Manjaro (Using docker compose setup)
2. OpenCTI version: 5.12.15
3. OpenCTI client: python 5.12.15 (and frontend)
4. Other environment details:

Reproducible Steps

1. Import the following example bundle taken from documentation

   {
   "id": "bundle--8fab937e-b694-41e3-b71c-0800271e87d2",
   "type": "bundle",
   "objects": [
   {
   "type": "email-message",
   "spec_version": "2.1",
   "id": "email-message--cf9b4b7f-14c8-5955-8065-020e0316b559",
   "is_multipart": true,
   "received_lines": [
   "from mail.example.com ([198.51.100.3]) by smtp.gmail.com with ESMTPSA id q23sm23309939wme.17.2016.07.19.07.20.32 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Jul 2016 07:20:40 -0700 (PDT)"
   ],
   "content_type": "multipart/mixed",
   "date": "2016-06-19T14:20:40.000Z",
   "from_ref": "email-addr--89f52ea8-d6ef-51e9-8fce-6a29236436ed",
   "to_refs": ["email-addr--d1b3bf0c-f02a-51a1-8102-11aba7959868"],
   "cc_refs": ["email-addr--e4ee5301-b52d-59cd-a8fa-8036738c7194"],
   "subject": "Check out this picture of a cat!",
   "additional_header_fields": {
   "Content-Disposition": "inline",
   "X-Mailer": "Mutt/1.5.23",
   "X-Originating-IP": "198.51.100.3"
   },
   "body_multipart": [
   {
     "content_type": "text/plain; charset=utf-8",
     "content_disposition": "inline",
     "body": "Cats are funny!"
   },
   {
     "content_type": "image/png",
     "content_disposition": "attachment; filename=\"tabby.png\"",
     "body_raw_ref": "artifact--4cce66f8-6eaa-53cb-85d5-3a85fca3a6c5"
   },
   {
     "content_type": "application/zip",
     "content_disposition": "attachment; filename=\"tabby_pics.zip\"",
     "body_raw_ref": "file--6ce09d9c-0ad3-5ebf-900c-e3cb288955b5"
   }
   ]
   },
   {
   "type": "email-addr",
   "spec_version": "2.1",
   "id": "email-addr--89f52ea8-d6ef-51e9-8fce-6a29236436ed",
   "value": "jdoe@example.com",
   "display_name": "John Doe"
   },
   {
   "type": "email-addr",
   "spec_version": "2.1",
   "id": "email-addr--d1b3bf0c-f02a-51a1-8102-11aba7959868",
   "value": "bob@example.com",
   "display_name": "Bob Smith"
   }, 
   {
   "type": "email-addr",
   "spec_version": "2.1",
   "id": "email-addr--e4ee5301-b52d-59cd-a8fa-8036738c7194",
   "value": "mary@example.com",
   "display_name": "Mary Smith"
   },
   {
   "type": "artifact",
   "spec_version": "2.1",
   "id": "artifact--4cce66f8-6eaa-53cb-85d5-3a85fca3a6c5",
   "mime_type": "image/jpeg",
   "payload_bin": "VBORw0KGgoAAAANSUhEUgAAADI==",
   "hashes": {
   "SHA-256": "effb46bba03f6c8aea5c653f9cf984f170dcdd3bbbe2ff6843c3e5da0e698766"
   }
   },
   {
   "type": "file",
   "spec_version": "2.1",
   "id": "file--6ce09d9c-0ad3-5ebf-900c-e3cb288955b5",
   "name": "tabby_pics.zip",
   "magic_number_hex": "504B0304",
   "hashes": {
   "SHA-256": "fe90a7e910cb3a4739bed9180e807e93fa70c90f25a8915476f5e4bfbac681db"
   }
   }
   ]
   }

2. Open workbench and validate it
3. Go to Observations
4. Open the "Email Message" observable with the subject "Check out this picture of a cat!"
5. Click on Knowledge

Expected Output

I expected there to be relations between the Email message and the mime components "tabby.png", "tabby_pics.zip" and the email mime component with text body, in the nested objects section.

Actual Output

The only nested objects are the email addresses contained in the bundle.

Additional information

I have tried to add the "Email MIME Component Type" to the stix bundle before sending it, but then it is not a valid bundle.

Screenshots (optional)

Failed result:

[Jipegien]

Seems we don't manage fully multi_body_part email as we do not create Parts mentionned in the "body_multipart". Problem is: if we respect the STIX 2.1, we have to use Email - Mime Part observables. It is very limited. A better way to manage this could be to create a new nested relationship (part-of_ref ?) and be able to link observables to the Email message. This way, we will leverage the full potential of our graph model. But we will need to handle this in the export connector, to be able to reconstruct the body_multipart content based on "part-of" nested relations.

[Jipegien]

@Kedae can we talk together about it regarding the complexity of the task plz?

[screencoffee]

Seems we don't manage fully multi_body_part email as we do not create Parts mentionned in the "body_multipart". Problem is: if we respect the STIX 2.1, we have to use Email - Mime Part observables. It is very limited. A better way to manage this could be to create a new nested relationship "part-of" and link artifact, file, etc. observables to the Email message. This way, we will leverage the full potential of our graph model. But we will need to handle this in the export connector, to be able to reconstruct the body_multipart content based on "part-of" nested relations.

This is kind of the workaround I did in my connector, but using a "related-to" relationship. I did try to use the "Email Message - MIME PART" observable type, inside OpenCTI but it doesn't seem like it supports "body_raw_ref" directly.

I think a good way to handle it would be to convert "body_multipart" into "Email Message - MIME PART" observables on import, because it preserves the "content-disposition". And as as you say recreate the property again on export.

[screencoffee]

Any updates on this? :)