Missing a schema for email

[krisATelastic]

Hey team, just working through the schema and there doesn't appear to be a defined schema for email in order to normalise logs from sources like sendmail/exim or devices like a Cisco ESA, etc...

Will something like this be coming in later cuts or would custom fields need to be utilised for this (i.e. create your own).

[zlammers]

Agreed, there should be email-specific fields. In our current schema, I had an email object with these fields:

email.source.source
email.source.user
email.source.domain
email.target.target
email.target.user
email.target.domain
email.subject
email.cc
email.bcc
email.content_type
email.reply_to

Though I came up with the above, I was never 100% happy with it due to:

1. The schema above assumes single target, which obviously is no good. Perhaps just simply retaining arrays of all targets, .cc's, and .bcc's is the better method.
2. The repitition of email.source.source and email.target.target -- I should have renamed them to ...email.source.original and email.target.original -- as the (user|domain) fields are created by Logstash off the email seen.

We would keyword all of them, except for subject, which we would retain both keyword and text versions.

[krisATelastic]

It's also handy to index things like the message Id, as if multiple SMTP relays are involved, you can also track the email flow as it comes in through say an SMTP proxy, right the way back to Exchange for example.

@MikePaquette what do you think on the above, I can see how we can structure the email addresses themselves, but do we have any guidelines for an email-type schema by chance?

[webmat]

Hey folks, I'd say email will definitely be coming at some point, although it hasn't been at the top of our priority list so far. So direct email support in the schema is not imminent either.

My recommendation is therefore to use custom fields for now. Providing expectations / desires / field ideas like @zlammers did above is extremely helpful in the meantime :-) This will help us come up solid out of the box support, when we get around to it.

Here are is another set of fields that was suggested elsewhere, around supporting emails in ECS. I'm recording it for future usage, but haven't had time to investigate more yet:

email.to 
email.from 
email.protocol 
email.sbrs 
email.cipher 
email.response 
email.mid 
email.icid 
email.dcid 
email.rid

[weichea]

+1. Is this being worked on?

For the to and from field, should we use source.user.email and destination.user.email instead?

[anhlqn]

I think a good reference for email fieldset is the Microsoft Exchange log fields https://docs.microsoft.com/en-us/exchange/mail-flow/transport-logs/message-tracking?view=exchserver-2019#fields-in-the-message-tracking-log-files

[webmat]

@weichea It's not being worked on at this time, unfortunately.

I think it more support around email makes sense, however. There's also a discussion around DMARC over here #593, if you're interested.

[jeffrey-e]

Would love to see e-mail fields being added, so +1 here!

[matbae8]

Would also love to see e-mail fields being added, so +1 here!

[mhunsber]

Not sure if this is related enough, or deserves its own issue, but an smtp schema similar to http/dns would also be useful. e.g.

smtp.response.status_code
smtp.request.body.content
smtp.response.body.content

[willemdh]

Had to index Exchange Message Tracking log recently. Parked it under an exchange object I made. This is the template I used:

PUT _template/ecs-exchange-message
{
  "order" : 2147483640,
  "index_patterns" : [
    "ecs-exchange-message-*"
  ],
  "settings" : {
    "index" : {
      "lifecycle" : {
        "name" : "ecs-exchange-message-ilm-policy",
        "rollover_alias" : "ecs-exchange-message-rollover"
      },
      "refresh_interval" : "30s",
      "analysis" : {
        "normalizer" : {
          "normalizer_lowercase" : {
            "filter" : [
              "lowercase"
            ],
            "type" : "custom"
          }
        }
      },
      "number_of_shards" : "1",
      "number_of_replicas" : "1"
    }
  },
  "mappings" : {
    "properties" : {
      "exchange" : {
        "properties" : {
          "message" : {
            "properties" : {
              "client" : {
                "properties" : {
                  "ip" : {
                    "type": "ip"
                  },
                  "domain" : {
                    "ignore_above": 1024,
                    "type": "keyword"
                  },
                  "original" : {
                    "properties" : {
                      "ip" : {
                        "type": "ip"
                      }
                    }
                  }
                }
              },
              "server" : {
                "properties" : {
                  "ip" : {
                    "type": "ip"
                  },
                  "domain" : {
                    "ignore_above": 1024,
                    "type": "keyword"
                  },
                  "original" : {
                    "properties" : {
                      "ip" : {
                        "type": "ip"
                      }
                    }
                  }
                }
              },
              "source" : {
                "properties" : {
                  "context" : {
                    "norms": false,
                    "type": "text"
                  },
                  "name" : {
                    "ignore_above": 1024,
                    "type": "keyword"
                  }
                }
              },
              "connector" : {
                "properties" : {
                  "id" : {
                    "ignore_above": 1024,
                    "type": "keyword"
                  }
                }
              },
              "event" : {
                "properties" : {
                  "code" : {
                    "ignore_above": 1024,
                    "type": "keyword"
                  }
                }
              },
              "internal" : {
                "properties" : {
                  "id" : {
                    "ignore_above": 1024,
                    "type": "keyword"
                  }
                }
              },
              "id" : {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "network" : {
                "properties" : {
                  "id" : {
                    "ignore_above": 1024,
                    "type": "keyword"
                  }
                }
              },
              "bytes" : {
                "type": "long"
              },
              "recipient" : {
                "properties" : {
                  "address" : {
                    "ignore_above": 1024,
                    "type": "keyword"
                  },
                  "status" : {
                    "ignore_above": 1024,
                    "type": "keyword"
                  },
                  "count" : {
                    "type": "integer"
                  },
                  "related" : {
                    "properties" : {
                      "address" : {
                        "ignore_above": 1024,
                        "type": "keyword"
                      }
                    }
                  }
                }
              },
              "reference" : {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "sender" : {
                "properties" : {
                  "address" : {
                    "ignore_above": 1024,
                    "type": "keyword"
                  }
                }
              },
              "tenant" : {
                "properties" : {
                  "id" : {
                    "ignore_above": 1024,
                    "type": "keyword"
                  }
                }
              },
              "subject": {
                "fields": {
                  "text": {
                    "norms": false,
                    "type": "text"
                  }
                },
                "ignore_above": 1024,
                "type": "keyword"
              },
              "return_path" : {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "info" : {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "directionality" : {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "data" : {
                "norms": false,
                "type": "text"
              },
              "transport" : {
                "properties" : {
                  "type" : {
                    "ignore_above": 1024,
                    "type": "keyword"
                  }
                }
              },
              "schema" : {
                "properties" : {
                  "version" : {
                    "ignore_above": 1024,
                    "type": "keyword"
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  "aliases" : {
    "exchange" : { },
    "exchange-message" : { },
    "exchange-message-tracking" : { }
  }
}

[willemdh]

@webmat Wondering what event.category and event.type I would have to choose from the allowed list for Exchange message tracking logs...? https://www.elastic.co/guide/en/ecs/current/ecs-allowed-values-event-category.html https://www.elastic.co/guide/en/ecs/current/ecs-allowed-values-event-type.html

event.category: mail event.type: info

Should there be an new event.category email?

[tnelftg]

There also needs to be a concept of has_attachment(s)

[webmat]

We created meta-issue #939 to discuss email support in ECS. Closing in favor of the meta issue.