Ingest Pipelines - Identify fields that need to be removed/added from the event or to the ES output

[roaksoax]

On a recent test of the m365_defender integration (slack thread), the user had to do to things that would make it appear that this feature of Logstash doesn't work out of the box.

The first, he needed to remove the @version field on the integration.

filter {
  elastic_integration {
    remove_field => ['@version']
    hosts => ["https://elasticsearch:9200/"]
    [...]
  }
}

And the second, is that he needed to set the _document_id_ field to the Elasticsearch Output, so that he wouldn't receive duplicate events.

output {
  elasticsearch {
    hosts => ["https://elasticsearch:9200/"]
    [...]
    document_id => "%{[@metadata][_ingest_document][id]}"
  }
}

The user had to do this because Logstash doesn't passthrough some metadata fields, and the issue has an RFC / potential solution.

However, this currently would cause users to try to use various integrations out of the box, and things would not work as expected, or would require them to add/remove fields/settings. This creates a barrier of entry and complicates the use of Logstash, specially, if multiple integrations were used that would require various use cases to be handled.

As such, at the very least, we need to identify and document the following:

• Fields that need to be removed per integration
• Fields that need to be added to the ES output

Integration	Fields to Remove	ES output fields to add
m365_defender	@version	document_id

[roaksoax]

Currently blocked on https://github.com/elastic/ingest-dev/issues/1792

[roaksoax]

In the latest version (starting from 0.0.2), we should not need to remove the @version field.

[andsel]

In the latest version (starting from 0.0.2), we should not need to remove the @version field.

I uses version 0.0.3 and not @version field was present in Logstash event, so no need to delete it.

[roaksoax]

Closing this issue as this is no longer a problem after https://github.com/elastic/ingest-dev/issues/1792 has been fixed and released.