Filebeat Elasticsearch ingest pipeline can't parse authentication messages for SIEM

[leweafan]

Describe the enhancement:

Filebeat Elasticsearch ingest pipeline can't parse failed authentication message. ECS fields important for SIEM like event.category, event.type, event.action, event.outcome and user.name are missing.

Failed authentication message should have fields:

• event.category = "authentication"
• event.action = "logon-failed"
• event.type = "start"
• event.outcome = "failure"
• user.name

Describe a specific use case for the enhancement or feature:

File /var/log/elasticsearch/elastic_server.json

{"type": "server", "timestamp": "2022-07-04T20:13:03,991+03:00", "level": "WARN", "component": "o.e.x.s.a.RealmsAuthenticator", "cluster.name": "elastic", "node.name": "elastic01", "message": "Authentication to realm default_native failed - Password authentication failed for xxx", "cluster.uuid": "1111111-1111111111_111", "node.id": "2222222222-2222222222" }

{"@timestamp":"2023-06-05T20:09:45.735Z", "log.level": "INFO", "message":"Authentication of [remote_monitoring_user] was terminated by realm [reserved] - failed to authenticate user [remote_monitoring_user]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elastic01][system_critical_read][T#4]","log.logger":"org.elasticsearch.xpack.security.authc.RealmsAuthenticator","elasticsearch.cluster.uuid":"1111111-1111111111_111","elasticsearch.node.id":"2222222222-2222222222","elasticsearch.node.name":"elastic01","elasticsearch.cluster.name":"elastic"}

--

File /var/log/elasticsearch/elastic.log

[2022-07-04T20:13:03,991][WARN ][o.e.x.s.a.RealmsAuthenticator] [elastic01] Authentication to realm default_native failed - Password authentication failed for xxx

[2023-06-05T23:09:45,735][INFO ][o.e.x.s.a.RealmsAuthenticator] [elastic01]Authentication of [remote_monitoring_user] was terminated by realm [reserved] - failed to authenticate user [remote_monitoring_user]

Steps to reproduce

POST /_ingest/pipeline/filebeat-7.17.4-elasticsearch-server-pipeline/_simulate
{
  "docs": [
    {
      "_source": {
        "@timestamp": "2022-07-04T20:13:03.991Z",
        "message": "{\"type\": \"server\", \"timestamp\": \"2022-07-04T20:13:03,991+03:00\", \"level\": \"WARN\", \"component\": \"o.e.x.s.a.RealmsAuthenticator\", \"cluster.name\": \"elastic\", \"node.name\": \"elastic01\", \"message\": \"Authentication to realm default_native failed - Password authentication failed for xxx\", \"cluster.uuid\": \"1111111-1111111111_111\", \"node.id\": \"2222222222-2222222222\" }"
      }
    }
  ]
}

POST /_ingest/pipeline/filebeat-7.17.4-elasticsearch-server-pipeline/_simulate
{
  "docs": [
    {
      "_source": {
        "@timestamp": "2022-07-04T20:13:03.991Z",
        "message": "[2022-07-04T20:13:03,991][WARN ][o.e.x.s.a.RealmsAuthenticator] [elastic01] Authentication to realm default_native failed - Password authentication failed for xxx"
      }
    }
  ]
}

Result

{
  "docs" : [
    {
      "doc" : {
        "_index" : "_index",
        "_type" : "_doc",
        "_id" : "_id",
        "_source" : {
          "@timestamp" : "2022-07-04T17:13:03.991Z",
          "elasticsearch" : {
            "server" : {
              "cluster" : { },
              "node" : { }
            },
            "cluster" : {
              "name" : "elastic",
              "uuid" : "1111111-1111111111_111"
            },
            "node" : {
              "name" : "elastic01",
              "id" : "2222222222-2222222222"
            },
            "component" : "o.e.x.s.a.RealmsAuthenticator"
          },
          "log" : {
            "level" : "WARN"
          },
          "host" : {
            "name" : "elastic01",
            "id" : "2222222222-2222222222"
          },
          "message" : "Authentication to realm default_native failed - Password authentication failed for xxx",
          "event" : {
            "ingested" : "2022-07-07T10:39:16.907749526Z",
            "category" : "database",
            "type" : "info",
            "created" : "2022-07-04T20:13:03.991Z",
            "kind" : "event"
          }
        },
        "_ingest" : {
          "timestamp" : "2022-07-07T10:39:16.907749526Z"
        }
      }
    }
  ]
}

{
  "docs" : [
    {
      "doc" : {
        "_index" : "_index",
        "_type" : "_doc",
        "_id" : "_id",
        "_source" : {
          "@timestamp" : "2022-07-04T20:13:03.991Z",
          "elasticsearch" : {
            "server" : { },
            "node" : {
              "name" : "elastic01"
            },
            "component" : "o.e.x.s.a.RealmsAuthenticator"
          },
          "log" : {
            "level" : "WARN"
          },
          "host" : {
            "name" : "elastic01"
          },
          "message" : "Authentication to realm default_native failed - Password authentication failed for xxx",
          "event" : {
            "ingested" : "2022-07-07T10:46:14.559537499Z",
            "category" : "database",
            "type" : "info",
            "created" : "2022-07-04T20:13:03.991Z",
            "kind" : "event"
          }
        },
        "_ingest" : {
          "timestamp" : "2022-07-07T10:46:14.559537499Z"
        }
      }
    }
  ]
}

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!