Some documents are not ingested into elasticsearch

[1337-42]

Logstash information:

Please include the following information:

1. Logstash version: 8.7.0
2. Logstash installation source: DEB
3. How is Logstash being run: systemd

Plugins installed: (bin/logstash-plugin list --verbose)

Using bundled JDK: /usr/share/logstash/jdk
logstash-codec-avro (3.4.0)
logstash-codec-cef (6.2.6)
logstash-codec-collectd (3.1.0)
logstash-codec-dots (3.0.6)
logstash-codec-edn (3.1.0)
logstash-codec-edn_lines (3.1.0)
logstash-codec-es_bulk (3.1.0)
logstash-codec-fluent (3.4.1)
logstash-codec-graphite (3.0.6)
logstash-codec-json (3.1.1)
logstash-codec-json_lines (3.1.0)
logstash-codec-line (3.1.1)
logstash-codec-msgpack (3.1.0)
logstash-codec-multiline (3.1.1)
logstash-codec-netflow (4.3.0)
logstash-codec-plain (3.1.0)
logstash-codec-rubydebug (3.1.0)
logstash-filter-aggregate (2.10.0)
logstash-filter-anonymize (3.0.6)
logstash-filter-cidr (3.1.3)
logstash-filter-clone (4.2.0)
logstash-filter-csv (3.1.1)
logstash-filter-date (3.1.15)
logstash-filter-de_dot (1.0.4)
logstash-filter-dissect (1.2.5)
logstash-filter-dns (3.2.0)
logstash-filter-drop (3.0.5)
logstash-filter-elasticsearch (3.13.0)
logstash-filter-fingerprint (3.4.2)
logstash-filter-geoip (7.2.13)
logstash-filter-grok (4.4.3)
logstash-filter-http (1.4.1)
logstash-filter-json (3.2.0)
logstash-filter-kv (4.7.0)
logstash-filter-memcached (1.1.0)
logstash-filter-metrics (4.0.7)
logstash-filter-mutate (3.5.6)
logstash-filter-prune (3.0.4)
logstash-filter-ruby (3.1.8)
logstash-filter-sleep (3.0.7)
logstash-filter-split (3.1.8)
logstash-filter-syslog_pri (3.2.0)
logstash-filter-throttle (4.0.4)
logstash-filter-translate (3.4.0)
logstash-filter-truncate (1.0.5)
logstash-filter-urldecode (3.0.6)
logstash-filter-useragent (3.3.3)
logstash-filter-uuid (3.0.5)
logstash-filter-xml (4.2.0)
logstash-input-azure_event_hubs (1.4.4)
logstash-input-beats (6.5.0)
└── logstash-input-elastic_agent (alias)
logstash-input-couchdb_changes (3.1.6)
logstash-input-dead_letter_queue (2.0.0)
logstash-input-elasticsearch (4.16.0)
logstash-input-exec (3.6.0)
logstash-input-file (4.4.4)
logstash-input-ganglia (3.1.4)
logstash-input-gelf (3.3.2)
logstash-input-generator (3.1.0)
logstash-input-graphite (3.0.6)
logstash-input-heartbeat (3.1.1)
logstash-input-http (3.6.1)
logstash-input-http_poller (5.4.0)
logstash-input-imap (3.2.0)
logstash-input-jms (3.2.2)
logstash-input-pipe (3.1.0)
logstash-input-redis (3.7.0)
logstash-input-snmp (1.3.1)
logstash-input-snmptrap (3.1.0)
logstash-input-stdin (3.4.0)
logstash-input-syslog (3.6.0)
logstash-input-tcp (6.3.2)
logstash-input-twitter (4.1.0)
logstash-input-udp (3.5.0)
logstash-input-unix (3.1.2)
logstash-integration-aws (7.1.0)
 ├── logstash-codec-cloudfront
 ├── logstash-codec-cloudtrail
 ├── logstash-input-cloudwatch
 ├── logstash-input-s3
 ├── logstash-input-sqs
 ├── logstash-output-cloudwatch
 ├── logstash-output-s3
 ├── logstash-output-sns
 └── logstash-output-sqs
logstash-integration-elastic_enterprise_search (2.2.1)
 ├── logstash-output-elastic_app_search
 └──  logstash-output-elastic_workplace_search
logstash-integration-jdbc (5.4.1)
 ├── logstash-input-jdbc
 ├── logstash-filter-jdbc_streaming
 └── logstash-filter-jdbc_static
logstash-integration-kafka (10.12.0)
 ├── logstash-input-kafka
 └── logstash-output-kafka
logstash-integration-rabbitmq (7.3.1)
 ├── logstash-input-rabbitmq
 └── logstash-output-rabbitmq
logstash-output-csv (3.0.8)
logstash-output-elasticsearch (11.13.1)
logstash-output-email (4.1.1)
logstash-output-file (4.3.0)
logstash-output-graphite (3.1.6)
logstash-output-http (5.5.0)
logstash-output-lumberjack (3.1.9)
logstash-output-nagios (3.0.6)
logstash-output-null (3.0.5)
logstash-output-pipe (3.0.6)
logstash-output-redis (5.0.0)
logstash-output-stdout (3.1.4)
logstash-output-tcp (6.1.1)
logstash-output-udp (3.2.0)
logstash-output-webhdfs (3.0.6)
logstash-patterns-core (4.3.4)

JVM (e.g. java -version):

Using bundled JDK: /usr/share/logstash/jdk

OS version (uname -a if on a Unix-like system):

Linux rsp-ingest 5.10.0-21-cloud-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64 GNU/Linux

Description of the problem including expected versus actual behavior: We are using Logstash for a project to extract data from a cluster and add it to a a second cluster. But after a few runs we noticed that in the destination cluster we are missing a few documents. Narrowing it down we found out that the documents that where missing all contained special characters like @ and # in the message field. The other few million documents are indexed as they should.

Steps to reproduce:

We use the following pipeline:

input {
  elasticsearch {
    cloud_id => "<REDACTED>"
    api_key => "<REDACTED>"
    ssl => true
    index => "logs-*"
    query => '{"query":{"bool":{"must":[],"filter":[{"bool":{"should":[{"bool":{"should":[{"term":{"agent.id":{"value":"2c468a0d-c853-4946-9d53-a5b1cc5a7530"}}}],"minimum_should_match":1}},{"bool":{"should":[{"term":{"agent.id":{"value":"6346c7a7-fecc-48e3-a9a6-812e1f9fe3aa"}}}],"minimum_should_match":1}},{"bool":{"should":[{"term":{"agent.id":{"value":"d9243148-23a5-4dd5-970c-f9e8c0deca94"}}}],"minimum_should_match":1}},{"bool":{"should":[{"term":{"agent.id":{"value":"1e9419b0-d4b2-40dc-bf92-8c6cc5babee9"}}}],"minimum_should_match":1}},{"bool":{"should":[{"term":{"agent.id":{"value":"c8eec386-ba96-4070-8e92-6f4a88a3074f"}}}],"minimum_should_match":1}},{"bool":{"should":[{"term":{"agent.id":{"value":"35c2f3d6-b57d-40c5-9116-f2123e7d0445"}}}],"minimum_should_match":1}},{"bool":{"should":[{"term":{"agent.id":{"value":"b9ecd31f-f985-47a2-a8c3-a7b093ed0e46"}}}],"minimum_should_match":1}},{"bool":{"should":[{"term":{"agent.id":{"value":"1362b81c-ab02-4ee4-9a7a-ba71f40db66b"}}}],"minimum_should_match":1}},{"bool":{"should":[{"term":{"agent.id":{"value":"23ec43cb-1204-4cac-858a-1bf620478a12"}}}],"minimum_should_match":1}},{"bool":{"should":[{"term":{"agent.id":{"value":"477344fa-413a-4bff-88e5-9542f02967e4"}}}],"minimum_should_match":1}},{"bool":{"should":[{"term":{"agent.id":{"value":"f2109900-90be-46ba-9442-1ecbeeaa972d"}}}],"minimum_should_match":1}},{"bool":{"should":[{"term":{"agent.id":{"value":"6826258b-3009-486a-b1a9-3a39511213ab"}}}],"minimum_should_match":1}}],"minimum_should_match":1}},{"range":{"@timestamp":{"format":"strict_date_optional_time","gte":"2023-01-24T10:17:41.292Z","lte":"2023-04-24T09:17:41.292Z"}}}],"should":[],"must_not":[]}}}'
    docinfo => true
    docinfo_target => "[@metadata][doc]"
    add_field => {
      identifier => "input"
    }
  }
}

filter {
  mutate {
    update => {"[data_stream][namespace]" => "<REDACTED>"}
  }
}

output {
  elasticsearch {
    cloud_id => "<REDACTED>"
    user => "<REDACTED>"
    password => "<REDACTED>"
    ssl => true
    data_stream => "true"
    data_stream_type => "logs"
    data_stream_dataset => "%{[data_stream][dataset]}"
    data_stream_namespace => "<REDACTED>"
    document_id => "%{[@metadata][doc][_id]}"
  }
}

All documents transfer just fine except for the ones that look like this:

{
  "_index": ".ds-logs-system.syslog-default-2023.03.31-000003",
  "_id": "nk5ZlIcBlMgfxDi-gpHW",
  "_version": 1,
  "_score": 0,
  "_source": {
    "agent": {
      "name": "<REDACTED>-e010a06d-4b81-482c-8e8b-738d9b7d64e4",
      "id": "2c468a0d-c853-4946-9d53-a5b1cc5a7530",
      "type": "filebeat",
      "ephemeral_id": "81c988c0-b914-47fe-b038-429cb0419c03",
      "version": "8.7.0"
    },
    "process": {
      "name": "kernel"
    },
    "log": {
      "file": {
        "path": "/var/log/syslog"
      },
      "offset": 0
    },
    "elastic_agent": {
      "id": "2c468a0d-c853-4946-9d53-a5b1cc5a7530",
      "version": "8.7.0",
      "snapshot": false
    },
    "message": "[    0.000000] Linux version 5.15.0-1018-gcp (buildd@lcy02-amd64-074) (gcc (Ubuntu 11.2.0-19ubuntu1) 11.2.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #24-Ubuntu SMP Thu Sep 8 07:14:47 UTC 2022 (Ubuntu 5.15.0-1018.24-gcp 5.15.53)",
    "cloud": {
      "availability_zone": "us-central1-a",
      "instance": {
        "name": "<REDACTED>-e010a06d-4b81-482c-8e8b-738d9b7d64e4",
        "id": "8146398734131082164"
      },
      "provider": "gcp",
      "service": {
        "name": "GCE"
      },
      "machine": {
        "type": "n2-standard-2"
      },
      "project": {
        "id": "<REDACTED>"
      },
      "account": {
        "id": "<REDACTED>"
      }
    },
    "input": {
      "type": "log"
    },
    "@timestamp": "2023-04-18T12:30:41.000Z",
    "system": {
      "syslog": {}
    },
    "ecs": {
      "version": "8.0.0"
    },
    "data_stream": {
      "namespace": "default",
      "type": "logs",
      "dataset": "system.syslog"
    },
    "host": {
      "hostname": "<REDACTED>-e010a06d-4b81-482c-8e8b-738d9b7d64e4",
      "os": {
        "kernel": "5.15.0-1018-gcp",
        "codename": "jammy",
        "name": "Ubuntu",
        "type": "linux",
        "family": "debian",
        "version": "22.04.1 LTS (Jammy Jellyfish)",
        "platform": "ubuntu"
      },
      "containerized": false,
      "ip": [
        "10.12.0.59",
        "fe80::4001:aff:fe0c:3b"
      ],
      "name": "<REDACTED>-e010a06d-4b81-482c-8e8b-738d9b7d64e4",
      "id": "b3b467b120019720a9d909465f942c51",
      "mac": [
        "42-01-0A-0C-00-3B"
      ],
      "architecture": "x86_64"
    },
    "event": {
      "agent_id_status": "verified",
      "ingested": "2023-04-18T12:31:46Z",
      "timezone": "+00:00",
      "dataset": "system.syslog"
    }
  },
  "fields": {
    "elastic_agent.version": [
      "8.7.0"
    ],
    "process.name.text": [
      "kernel"
    ],
    "host.os.name.text": [
      "Ubuntu"
    ],
    "host.hostname": [
      "<REDACTED>-e010a06d-4b81-482c-8e8b-738d9b7d64e4"
    ],
    "host.mac": [
      "42-01-0A-0C-00-3B"
    ],
    "cloud.availability_zone": [
      "us-central1-a"
    ],
    "host.ip": [
      "10.12.0.59",
      "fe80::4001:aff:fe0c:3b"
    ],
    "cloud.instance.id": [
      "8146398734131082164"
    ],
    "agent.type": [
      "filebeat"
    ],
    "event.module": [
      "system"
    ],
    "host.os.version": [
      "22.04.1 LTS (Jammy Jellyfish)"
    ],
    "host.os.kernel": [
      "5.15.0-1018-gcp"
    ],
    "host.os.name": [
      "Ubuntu"
    ],
    "agent.name": [
      "<REDACTED>-e010a06d-4b81-482c-8e8b-738d9b7d64e4"
    ],
    "host.name": [
      "<REDACTED>-e010a06d-4b81-482c-8e8b-738d9b7d64e4"
    ],
    "elastic_agent.snapshot": [
      false
    ],
    "event.agent_id_status": [
      "verified"
    ],
    "host.id": [
      "b3b467b120019720a9d909465f942c51"
    ],
    "event.timezone": [
      "+00:00"
    ],
    "host.os.type": [
      "linux"
    ],
    "elastic_agent.id": [
      "2c468a0d-c853-4946-9d53-a5b1cc5a7530"
    ],
    "data_stream.namespace": [
      "default"
    ],
    "host.os.codename": [
      "jammy"
    ],
    "input.type": [
      "log"
    ],
    "log.offset": [
      0
    ],
    "message": [
      "[    0.000000] Linux version 5.15.0-1018-gcp (buildd@lcy02-amd64-074) (gcc (Ubuntu 11.2.0-19ubuntu1) 11.2.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #24-Ubuntu SMP Thu Sep 8 07:14:47 UTC 2022 (Ubuntu 5.15.0-1018.24-gcp 5.15.53)"
    ],
    "data_stream.type": [
      "logs"
    ],
    "host.architecture": [
      "x86_64"
    ],
    "process.name": [
      "kernel"
    ],
    "cloud.provider": [
      "gcp"
    ],
    "cloud.machine.type": [
      "n2-standard-2"
    ],
    "event.ingested": [
      "2023-04-18T12:31:46.000Z"
    ],
    "@timestamp": [
      "2023-04-18T12:30:41.000Z"
    ],
    "agent.id": [
      "2c468a0d-c853-4946-9d53-a5b1cc5a7530"
    ],
    "cloud.service.name": [
      "GCE"
    ],
    "cloud.account.id": [
      "<REDACTED>"
    ],
    "host.os.platform": [
      "ubuntu"
    ],
    "host.containerized": [
      false
    ],
    "ecs.version": [
      "8.0.0"
    ],
    "log.file.path": [
      "/var/log/syslog"
    ],
    "data_stream.dataset": [
      "system.syslog"
    ],
    "agent.ephemeral_id": [
      "81c988c0-b914-47fe-b038-429cb0419c03"
    ],
    "agent.version": [
      "8.7.0"
    ],
    "host.os.family": [
      "debian"
    ],
    "event.dataset": [
      "system.syslog"
    ],
    "cloud.project.id": [
      "<REDACTED>"
    ],
    "cloud.instance.name": [
      "<REDACTED>-e010a06d-4b81-482c-8e8b-738d9b7d64e4"
    ]
  }
}

The logstash logs don't provined any warning or errors.

cc: @Aegrah

[yaauie]

My best guess is that an active Elastic integration is intercepting the event and mutating it beyond recognition.

1. how are you searching for the event to determine that it is missing?
2. what integrations are active in Elasticsearch? if the System integration is active, are the events intentionaly being handled by the system.syslog ingest pipeline? This ingest pipeline certainly not handle the pasted event as described below:
   • a grok processor tries (and fails) to map the contents of message through any of three patterns, none of which will match
   • the pipeline's on_failure will set the error.message field and cause processing to continue
   • the next processor will perform an unguarded removal of the event's message (which would make it difficult to search for)
   • the next processor will attempt to rename system.syslog.message to message, but will not do so because system.syslog.message on this event is not set.
   • the next processor will attempt to parse the date from system.syslog.timestamp, but will fail to do so because that field is not set on this event, causing it to append more to error.message
   • and so on.
3. it is possible to bypass an event's default ingest pipeline (as configured by the active integrations) by explicitly configuring the Logstash ES output with pipeline => "_none".

[Aegrah]

@yaauie thanks for the helpful response! We ran a quick test, running the pipeline with the suggested pipeline => "_none" flag and the data was succesfuly shipped.

To make sure it was not a fluke, we ran the same logstash pipeline without pipeline => "_none", which once again failed to ship the data. During this test, the datastream that Logstash was supposed to ship to was created, but after looking in Kibana logs-* filtering on the added field, we found 0 hits. Looking in dev tools, we can kind of see what is going on:

GET .ds-logs-system.syslog-logstash_test-2023.04.26-000001/_doc/ayujlIcBbdM3iBUU354C

{
  "_index": ".ds-logs-system.syslog-logstash_test-2023.04.26-000001",
  "_id": "ayujlIcBbdM3iBUU354C",
  "_version": 1,
  "_seq_no": 251,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "identifier": "logstash_test",
    "process": {
      "name": "kernel"
    },
    "agent": {
      "name": "[...REDACTED...]",
      "id": "[...REDACTED...]",
      "type": "filebeat",
      "ephemeral_id": "[...REDACTED...]",
      "version": "8.7.0"
    },
    "log": {
      "file": {
        "path": "/var/log/syslog"
      },
      "offset": 10324
    },
    "elastic_agent": {
      "id": "[...REDACTED...]",
      "version": "8.7.0",
      "snapshot": false
    },
    "message": "UTC 2022 (Ubuntu 5.15.0-1018.24-gcp 5.15.53)",
    "cloud": {
      "availability_zone": "[...REDACTED...]",
      "instance": {
        "name": "[...REDACTED...]",
        "id": "[...REDACTED...]"
      },
      "provider": "[...REDACTED...]",
      "machine": {
        "type": "[...REDACTED...]"
      },
      "service": {
        "name": "[...REDACTED...]"
      },
      "project": {
        "id": "[...REDACTED...]"
      },
      "account": {
        "id": "[...REDACTED...]"
      }
    },
    "input": {
      "type": "log"
    },
    "@timestamp": "2023-09-08T07:14:47.000Z",
    "system": {
      "syslog": {}
    },
    "ecs": {
      "version": "8.0.0"
    },
    "data_stream": {
      "namespace": "logstash_test",
      "type": "logs",
      "dataset": "system.syslog"
    },
    "host": {
      "hostname": "[...REDACTED...]",
      "os": {
        "kernel": "5.15.0-1018-gcp",
        "codename": "jammy",
        "name": "Ubuntu",
        "family": "debian",
        "type": "linux",
        "version": "22.04.1 LTS (Jammy Jellyfish)",
        "platform": "ubuntu"
      },
      "ip": [
        "10.12.0.30",
        "fe80::4001:aff:fe0c:1e"
      ],
      "containerized": false,
      "name": "[...REDACTED...]",
      "id": "[...REDACTED...]",
      "mac": [
        "[...REDACTED...]"
      ],
      "architecture": "x86_64"
    },
    "@version": "1",
    "event": {
      "agent_id_status": "auth_metadata_missing",
      "ingested": "2023-04-26T07:10:06Z",
      "timezone": "+00:00",
      "dataset": "system.syslog"
    }
  }
}

The message field contains only a part of the ingested string "message": "UTC 2022 (Ubuntu 5.15.0-1018.24-gcp 5.15.53)", while the full string would look like [ 0.000000] Linux version 5.15.0-1018-gcp (buildd@lcy02-amd64-074) (gcc (Ubuntu 11.2.0-19ubuntu1) 11.2.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #24-Ubuntu SMP Thu Sep 8 07:14:47 UTC 2022 (Ubuntu 5.15.0-1018.24-gcp 5.15.53).

A search in logs-* on message : "UTC 2022 (Ubuntu 5.15.0-1018.24-gcp 5.15.53)" or error.message: "UTC 2022 (Ubuntu 5.15.0-1018.24-gcp 5.15.53)" retrieves 0 results.

Somehow it turns these 254 docs into unsearchable docs. Looking at the mappings for this index, it seems like there shouldn't be any index / search issues.

GET /.ds-logs-system.syslog-logstash_test-2023.04.26-000001/_mapping

{
  ".ds-logs-system.syslog-logstash_test-2023.04.26-000001": {
    "mappings": {
      "_meta": {
        "managed_by": "fleet",
        "managed": true,
        "package": {
          "name": "system"
        }
      },
      "_data_stream_timestamp": {
        "enabled": true
      },
      "dynamic_templates": [
        {
          "container.labels": {
            "path_match": "container.labels.*",
            "match_mapping_type": "string",
            "mapping": {
              "type": "keyword"
            }
          }
        },
        {
          "strings_as_keyword": {
            "match_mapping_type": "string",
            "mapping": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        }
      ],
      "date_detection": false,
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "@version": {
          "type": "keyword",
          "ignore_above": 1024
        },
        "agent": {
          "properties": {
            "ephemeral_id": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "id": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "name": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "type": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "version": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "cloud": {
          "properties": {
            "account": {
              "properties": {
                "id": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "availability_zone": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "image": {
              "properties": {
                "id": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "instance": {
              "properties": {
                "id": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "name": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "machine": {
              "properties": {
                "type": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "project": {
              "properties": {
                "id": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "provider": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "region": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "service": {
              "properties": {
                "name": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            }
          }
        },
        "container": {
          "properties": {
            "id": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "image": {
              "properties": {
                "name": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "name": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "data_stream": {
          "properties": {
            "dataset": {
              "type": "constant_keyword",
              "value": "system.syslog"
            },
            "namespace": {
              "type": "constant_keyword",
              "value": "logstash_test"
            },
            "type": {
              "type": "constant_keyword",
              "value": "logs"
            }
          }
        },
        "ecs": {
          "properties": {
            "version": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "elastic_agent": {
          "properties": {
            "id": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "snapshot": {
              "type": "boolean"
            },
            "version": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "event": {
          "properties": {
            "action": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "agent_id_status": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "category": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "code": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "created": {
              "type": "date"
            },
            "dataset": {
              "type": "constant_keyword",
              "value": "system.syslog"
            },
            "ingested": {
              "type": "date",
              "format": "strict_date_time_no_millis||strict_date_optional_time||epoch_millis"
            },
            "kind": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "module": {
              "type": "constant_keyword",
              "value": "system"
            },
            "outcome": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "provider": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "sequence": {
              "type": "long"
            },
            "timezone": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "type": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "fingerprint": {
          "type": "keyword",
          "fields": {
            "text": {
              "type": "text"
            }
          }
        },
        "host": {
          "properties": {
            "architecture": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "containerized": {
              "type": "boolean"
            },
            "domain": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "hostname": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "id": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "ip": {
              "type": "ip"
            },
            "mac": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "name": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "os": {
              "properties": {
                "build": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "codename": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "family": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "full": {
                  "type": "keyword",
                  "ignore_above": 1024,
                  "fields": {
                    "text": {
                      "type": "match_only_text"
                    }
                  }
                },
                "kernel": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "name": {
                  "type": "keyword",
                  "ignore_above": 1024,
                  "fields": {
                    "text": {
                      "type": "text"
                    }
                  }
                },
                "platform": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "type": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "version": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "type": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "identifier": {
          "type": "keyword",
          "ignore_above": 1024
        },
        "input": {
          "properties": {
            "type": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "log": {
          "properties": {
            "file": {
              "properties": {
                "path": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "offset": {
              "type": "long"
            }
          }
        },
        "message": {
          "type": "match_only_text"
        },
        "process": {
          "properties": {
            "name": {
              "type": "keyword",
              "ignore_above": 1024,
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              }
            },
            "pid": {
              "type": "long"
            }
          }
        },
        "system": {
          "properties": {
            "syslog": {
              "type": "object"
            }
          }
        }
      }
    }
  }
}

The _settings also look normal. Any clue how this could be caused?

Other than that, thanks for the solution, as your suggestion did fix the issue.

[1337-42]

@yaauie thanks for the support!