[cisco_secure_email_gateway] Missing parsers for some mail_logs category messages.

[leandrojmp]

Hello,

On the integration for Cisco Secure Email Gateway we need to use the value of the field email.message_id to be able to correlate multiple events, however there are a lot of log messages in the field cisco_secure_email_gateway.log.message that currently are not being parsed to extract this information and this impacts or analysis.

For example, messages with information for DKIM and DMARC are not being parsed and there are a lot of useful information there.

This is a sample of some messages related to the same email message id that are not being parsed to extract the value for the field email.message_id.

MID 68119155 RID [0] Response '2.0.0 OK  1687954632 redactedstring - gsmtp'
Message finished MID 68119155 done
MID 68119155 interim verdict using engine: CASE bulk
MID 68119155 interim verdict using engine: CASE marketing
MID 68119155 interim AV verdict using Sophos CLEAN
MID 68119155 AMP file reputation verdict : SKIPPED (no attachment in message)
MID 68119155 using engine: GRAYMAIL marketing_mail
MID 68119155 using engine: GRAYMAIL positive
MID 68119155 Outbreak Filters: verdict negative
MID 68119155 queued for delivery
MID 68119155 using engine: SPF Verdict Cache using cached verdict
MID 68119155 SPF: mailfrom identity no-reply@redactedMailFrom.com Pass (v=spf1)
MID 68119155 DKIM: pass signature verified (d=redactedMailFrom.com s=srsa2048 i=@redactedMailFrom.com)
MID 68119155 DMARC: Message from domain redactedMailFrom.com, DMARC pass (SPF aligned True, DKIM aligned True)
MID 68119155 DMARC: Verification passed
MID 68119155 Message-ID '<redacted@redactedMailFrom.com>'
MID 68119155 Subject "redacted subject"
MID 68119155 SDR: Domains for which SDR is requested: reverse DNS host: redacted.redactedMailFrom.com, helo: redacted.redactedMailFrom.com, env-from: redactedMailFrom.com, header-from: redactedMailFrom.com, reply-to: redactedMailFrom.com
MID 68119155 SDR: Consolidated Sender Threat Level: Neutral, Threat Category: N/A, Suspected Domain(s) : N/A (other reasons for verdict). Sender Maturity: 30 days (or greater) for domain: redacted.redactedMailFrom.com
MID 68119155 SDR: Tracker Header : redactedTrackerHeader
MID 68119155 matched all recipients for per-recipient policy DEFAULT in the inbound table
MID 68119155 SDR: Domains for which SDR is requested: reverse DNS host: redacted.redactedMailFrom.com, helo: redacted.redactedMailFrom.com, env-from: redactedMailFrom.com, header-from: Not Present, reply-to: Not Present
MID 68119155 SDR: Consolidated Sender Threat Level: Neutral, Threat Category: N/A, Suspected Domain(s) : N/A (other reasons for verdict). Sender Maturity: 30 days (or greater) for domain: redacted.redactedMailFrom.com

None of those messages are being parsed.

As a quick workaround I'm using the following processor on the custom pipeline to catch all the majority of them:

{
    "dissect": {
      "field": "cisco_secure_email_gateway.log.message",
      "pattern": "MID %{email.message_id} %{}",
      "ignore_missing": true,
      "if": "ctx.email?.message_id == null && ctx.cisco_secure_email_gateway?.log?.category?.name == \"mail_logs\" && ctx.cisco_secure_email_gateway?.log?.message.startsWith(\"MID\")",
      "ignore_failure": true
    }
}

But there are more information that could be extracted from those messages.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[jamiehynds]

Thanks @leandrojmp - we have enough data here to make updates to our pipeline. We've added this issue to our backlog and @piyush-elastic will be in touch if we have any questions when we work on the update.

[jamiehynds]

@leandrojmp Secure Email integration update (v1.18) also includes support for these logs too.