[META] Implement ECS-Compatibility Mode in Bundled Plugins

yaauie commented 4 years ago

Plugins that are bundled with Logstash need to be able to run in an ECS-Compatible mode by default in an upcoming major release of Logstash, unless explicitly configured to do otherwise.

To determine the scope of changes needed to implement an ECS-Compatibility mode, a Logstash plugin will need to be categorized based on its implicit behavour to determine the effort to bring their defaults into ECS compliance:

Exclusively uses fields that align with ECS (no ECS-Compatibility mode needed)

Uses one or more fields that conflict with ECS (ECS-Compatibility mode required)

Uses one or more fields that are undefined in the latest ECS and therefore at risk of future conflict (ECS-Compatibility mode recommended, aiming to minimize this risk)

-- https://github.com/elastic/logstash/issues/11623

Below is our list of bundled plugins. The list will be kept up-to-date with links issues on the individual projects, along with one of the three categories (align, conflict, undefined).

Input Plugins

Input Plugins are generally reliant on Codec Plugins to produce their event structure, but many add metadata related to the source of the input (e.g., sender host info for inbound connections, file paths for discovered files, etc.), and some include embedded Filter Plugins that must separately implement ECS-Compatibility.

[x] Tier 1 Supported Inputs
- [x] ~Azure Event Hubs Input - https://github.com/logstash-plugins/logstash-input-azure_event_hubs/issues/49~
- [x] Beats Input - https://github.com/logstash-plugins/logstash-input-beats/issues/387
- [x] Elasticsearch Input - https://github.com/logstash-plugins/logstash-input-elasticsearch/issues/117
- [x] File Input - https://github.com/logstash-plugins/logstash-input-file/issues/259
- [x] Graphite Input - https://github.com/logstash-plugins/logstash-input-graphite/issues/19
- [x] Heartbeat Input - https://github.com/logstash-plugins/logstash-input-heartbeat/issues/17
- [x] HTTP Input - https://github.com/logstash-plugins/logstash-input-http/issues/123
- [x] HTTP Poller Input - https://github.com/logstash-plugins/logstash-input-http_poller/issues/117
- [x] JDBC Input* - https://github.com/logstash-plugins/logstash-integration-jdbc/issues/19
- [x] Kafka Input* - https://github.com/logstash-plugins/logstash-integration-kafka/issues/17
- [x] RabbitMQ Input* - https://github.com/logstash-plugins/logstash-integration-rabbitmq/issues/28
- [x] ~Redis Input - https://github.com/logstash-plugins/logstash-input-redis/issues/78~ (_no field names used except [@metadata][redis_channel]_)
- [x] S3 Input - https://github.com/logstash-plugins/logstash-input-s3/issues/196
- [x] Stdin Input - https://github.com/logstash-plugins/logstash-input-stdin/issues/18
- [x] Syslog Input - https://github.com/logstash-plugins/logstash-input-syslog/issues/60
- [x] TCP Input - https://github.com/logstash-plugins/logstash-input-tcp/issues/149
- [x] UDP Input - https://github.com/logstash-plugins/logstash-input-udp/issues/48
[ ] Tier 2 Supported Inputs

Codec Plugins

Codec Plugins produce Events from a sequence of Bytes, and are responsible for the basic structure of those Events, and while some implicitly take the produced structure verbatim from the deserialized input (e.g., JSON*), others decode the bytes they are given into their own structure (e.g., CEF).

[x] Tier 1 Supported Codecs
- [x] Avro Codec - https://github.com/logstash-plugins/logstash-codec-avro/issues/34
- [x] CEF Codec - https://github.com/logstash-plugins/logstash-codec-cef/issues/77
- [x] CSV Codec (new) - https://github.com/logstash-plugins/logstash-codec-csv/issues/5
- [x] ES Bulk Codec - https://github.com/logstash-plugins/logstash-codec-es_bulk/issues/19
- [x] JSON Codec - https://github.com/logstash-plugins/logstash-codec-json/issues/36
- [x] JSON Lines Codec - https://github.com/logstash-plugins/logstash-codec-json_lines/issues/39
- [x] Line Codec - https://github.com/logstash-plugins/logstash-codec-line/issues/17
- [x] Multiline Codec - https://github.com/logstash-plugins/logstash-codec-multiline/issues/66
- [x] Plain Codec - https://github.com/logstash-plugins/logstash-codec-plain/issues/9
- [x] ~RubyDebug Codec~ (output only)
[x] Tier 2 Supported Codecs
- [x] ~Dots Codec~ (output only)
- [x] ~~EDN Codec~~ event_factory + target only
- [x] ~~EDN Lines Codec~~ event_factory + target only
- [x] ~~Fluent Codec~~ event_factory + target only
- [x] ~~Graphite Codec~~ event_factory only
- [x] ~~MsgPack Codec~~ event_factory + target only
- [x] Netflow Codec (k:properly isolated under target => 'netflow' by default)
- [x] ~~Collectd Codec~~ event_factory + target only

Filter Plugins

Filter plugins manipulate Events by reading and writing from their fields. To be ECS-Compatible, a Filter must not read from or write to ECS-conflicting fields unless the field name is explicitly given in the plugin's configuration.

[x] Tier 1 Supported Filters
- [x] ~CIDR Filter~ (explicit field names required)
- [x] Clone Filter - https://github.com/logstash-plugins/logstash-filter-clone/issues/21
- [x] CSV Filter - https://github.com/logstash-plugins/logstash-filter-csv/issues/80
- [x] ~Date Filter~ (implicit defaults are ECS-Compatible)
- [x] ~Dissect Filter~ (explicit field names required)
- [x] ~DNS Filter~ (explicit field names required)
- [x] ~Drop Filter~ (no field names used)
- [x] ~Elasticsearch Filter~ (explicit field names required)
- [x] Grok Filter - https://github.com/logstash-plugins/logstash-filter-grok/issues/157
- [x] HTTP Filter - https://github.com/logstash-plugins/logstash-filter-http/issues/22
- [x] Fingerprint Filter - https://github.com/logstash-plugins/logstash-filter-fingerprint/issues/54
- [x] Geoip Filter - https://github.com/logstash-plugins/logstash-filter-geoip/issues/163
- [x] JDBC Static Filter* - https://github.com/logstash-plugins/logstash-integration-jdbc/issues/19
- [x] JDBC Streaming Filter* - https://github.com/logstash-plugins/logstash-integration-jdbc/issues/19
- [x] JSON Filter - https://github.com/logstash-plugins/logstash-filter-json/issues/45
- [x] KV Filter - https://github.com/logstash-plugins/logstash-filter-kv/issues/90
- [x] ~Memcached Filter~ (explicit field names required)
- [x] ~Mutate Filter~ (explicit field names required)
- [x] ~Prune Filter~ (explicit field names required)
- [x] ~Ruby Filter~ (no implicit field names used)
- [x] ~Sleep Filter~ (no field names used)
- [x] ~Split Filter~ (no implicit field names used)
- [x] Syslog_pri Filter - https://github.com/logstash-plugins/logstash-filter-syslog_pri/issues/8
- [x] Translate Filter - https://github.com/logstash-plugins/logstash-filter-translate/issues/86
- [x] ~Truncate Filter~ (no implicit field names used)
- [x] ~Urldecode Filter~ (no conflicting implicit field names used)
- [x] Useragent Filter - https://github.com/logstash-plugins/logstash-filter-useragent/issues/66
- [x] ~UUID Filter~ (no implicit field names used)
- [x] ~XML Filter~ (no implicit field names used)
[ ] Tier 2 Supported Filters
- [x] ~Aggregate Filter~ (k:user-driven configuration)
- [x] ~Anonymize Filter~ (k:deprecated since 2017 - replaced w fingerprint filter)
- [x] De_dot Filter (k: simply removes '.', using nested => true could improve ECS compatibility)
- [ ] Metrics Filter https://github.com/logstash-plugins/logstash-filter-metrics/issues/60 (noop for now?)
- [x] ~Throttle Filter~ (k: key => ... selector is user arbitrary)

Output Plugins

Output Plugins do not generally manipulate event structure, but are included in this list for completeness as some may benefit from an ECS-Compatibility mode (e.g. Elasticsearch's template management).