ECS deprecation warnings when ECS is disabled on all pipelines in pipelines.yml

[nicenemo]

Logstash information:

Please include the following information:

1. Logstash version (e.g. bin/logstash --version)

Using bundled JDK: /usr/share/logstash/jdk

logstash 7.17.2

2. Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker)

apt from Elastic repositories.

3. How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes)

Systemd

Plugins installed: (bin/logstash-plugin list --verbose)

Using bundled JDK: /usr/share/logstash/jdk
logstash-codec-avro (3.3.1)
logstash-codec-cef (6.2.4)
logstash-codec-collectd (3.1.0)
logstash-codec-dots (3.0.6)
logstash-codec-edn (3.1.0)
logstash-codec-edn_lines (3.1.0)
logstash-codec-es_bulk (3.1.0)
logstash-codec-fluent (3.4.1)
logstash-codec-graphite (3.0.6)
logstash-codec-json (3.1.0)
logstash-codec-json_lines (3.1.0)
logstash-codec-line (3.1.1)
logstash-codec-msgpack (3.1.0)
logstash-codec-multiline (3.1.1)
logstash-codec-netflow (4.2.2)
logstash-codec-plain (3.1.0)
logstash-codec-rubydebug (3.1.0)
logstash-filter-aggregate (2.10.0)
logstash-filter-anonymize (3.0.6)
logstash-filter-cidr (3.1.3)
logstash-filter-clone (4.2.0)
logstash-filter-csv (3.1.1)
logstash-filter-date (3.1.14)
logstash-filter-de_dot (1.0.4)
logstash-filter-dissect (1.2.5)
logstash-filter-dns (3.1.4)
logstash-filter-drop (3.0.5)
logstash-filter-elasticsearch (3.11.1)
logstash-filter-fingerprint (3.3.2)
logstash-filter-geoip (7.2.12)
logstash-filter-grok (4.4.1)
logstash-filter-http (1.2.1)
logstash-filter-json (3.2.0)
logstash-filter-kv (4.5.0)
logstash-filter-memcached (1.1.0)
logstash-filter-metrics (4.0.7)
logstash-filter-multiline (3.0.4)
logstash-filter-mutate (3.5.6)
logstash-filter-prune (3.0.4)
logstash-filter-ruby (3.1.8)
logstash-filter-sleep (3.0.7)
logstash-filter-split (3.1.8)
logstash-filter-syslog_pri (3.1.1)
logstash-filter-throttle (4.0.4)
logstash-filter-translate (3.3.0)
logstash-filter-truncate (1.0.5)
logstash-filter-urldecode (3.0.6)
logstash-filter-useragent (3.3.3)
logstash-filter-uuid (3.0.5)
logstash-filter-xml (4.1.3)
logstash-input-azure_event_hubs (1.4.3)
logstash-input-beats (6.2.6)
└── logstash-input-elastic_agent (alias)
logstash-input-couchdb_changes (3.1.6)
logstash-input-dead_letter_queue (1.1.11)
logstash-input-elasticsearch (4.12.2)
logstash-input-exec (3.4.0)
logstash-input-file (4.4.0)
logstash-input-ganglia (3.1.4)
logstash-input-gelf (3.3.1)
logstash-input-generator (3.1.0)
logstash-input-graphite (3.0.6)
logstash-input-heartbeat (3.1.1)
logstash-input-http (3.4.5)
logstash-input-http_poller (5.1.0)
logstash-input-imap (3.2.0)
logstash-input-jms (3.2.1)
logstash-input-pipe (3.1.0)
logstash-input-redis (3.7.0)
logstash-input-s3 (3.8.3)
logstash-input-snmp (1.3.1)
logstash-input-snmptrap (3.1.0)
logstash-input-sqs (3.1.3)
logstash-input-stdin (3.4.0)
logstash-input-syslog (3.6.0)
logstash-input-tcp (6.2.7)
logstash-input-twitter (4.1.0)
logstash-input-udp (3.5.0)
logstash-input-unix (3.1.1)
logstash-integration-elastic_enterprise_search (2.1.2)
 ├── logstash-output-elastic_app_search
 └──  logstash-output-elastic_workplace_search
logstash-integration-jdbc (5.2.4)
 ├── logstash-input-jdbc
 ├── logstash-filter-jdbc_streaming
 └── logstash-filter-jdbc_static
logstash-integration-kafka (10.9.0)
 ├── logstash-input-kafka
 └── logstash-output-kafka
logstash-integration-rabbitmq (7.3.0)
 ├── logstash-input-rabbitmq
 └── logstash-output-rabbitmq
logstash-output-cloudwatch (3.0.10)
logstash-output-csv (3.0.8)
logstash-output-elasticsearch (11.4.1)
logstash-output-email (4.1.1)
logstash-output-file (4.3.0)
logstash-output-graphite (3.1.6)
logstash-output-http (5.2.5)
logstash-output-lumberjack (3.1.9)
logstash-output-nagios (3.0.6)
logstash-output-null (3.0.5)
logstash-output-pipe (3.0.6)
logstash-output-redis (5.0.0)
logstash-output-s3 (4.3.5)
logstash-output-sns (4.0.8)
logstash-output-sqs (6.0.0)
logstash-output-stdout (3.1.4)
logstash-output-tcp (6.0.2)
logstash-output-udp (3.2.0)
logstash-output-webhdfs (3.0.6)
logstash-patterns-core (4.3.2)

JVM (e.g. java -version): bundled with Logstash

OS version (uname -a if on a Unix-like system):

Linux elk 5.10.0-13-cloud-arm64 #1 SMP Debian 5.10.106-1 (2022-03-17) aarch64 GNU/Linux

Description of the problem including expected versus actual behavior:

We have data that adheres to a strict schema but will never be ECS compliant therefore we disabled ECS compatibility on a per pipeline basis in pipelines.yml. Still we get deprecation warning on ECS compatibility. We did not configure ECS compatibility within the pipeline definitions. We are NOT using Elasticsearch data streams for these pipelines as output. we do use Elasticsearch ILM as output and file output. Other input/outputs are pipeline and sqs.

Still we see depreaction warnings.

Steps to reproduce:

Have one or more pipeline definitions with ecs compatibility set to disabled in pipelines.yml:

---
- pipeline.id: "my_pipeline"
  path.config: "/etc/logstash/pipelines/my_pipeline.conf"
  pipeline.ecs_compatibility: disabled
  pipeline.workers: 2
  queue.type: persisted
  queue.checkpoint.writes: 1
  queue.max_bytes: 4gb
  path.queue: "/opt/logstash_queues/my_pipeline"
  path.dead_letter_queue: "/opt/logstash_queues/my_pipeline.deadletter"
  dead_letter_queue.max_bytes: 2gb
- pipeline.id "my_other_pipeline"
  ....

All pipelines in the pipelines.yml are defines with ECS compatibility disabled.

For now I did not include all pipeline definitions and configs because redacting those is a lot of work and I don't think adding those will add much.

Provide logs (if relevant):

[2022-04-07T08:19:36,839][WARN ][deprecation.logstash.codecs.json][my_pipeline] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.

[yaauie]

I have traced this specific issue down to the HTTP input plugin's content-type-to-codec mapping (the additional_codecs option, including its default value).

The HTTP input instantiates codec instances for its additional_codecs option directly, without propagating the execution context. This means that those codecs do not have access to the pipeline settings, which in turn makes the deprecation noise legitimate -- if this pipeline were to be migrated to Logstash 8, the behaviour of those no-context codecs would use the default setting which would be a breaking change to the pipeline.

I have three paths forward for you:

Use updated HTTP input plugin

Version 3.5.1 released today correctly contextualizes the inner codecs so that they can respect the pipeline setting (https://github.com/logstash-plugins/logstash-input-http/pull/152).

bin/logstash-plugin update logstash-input-http

Which should include the output:

Updated logstash-input-http 3.4.5 to 3.5.1

Avoid use of additional_codecs

If your HTTP endpoint only uses a single content type, you can bypass the problematic content-type-to-codec mapping and use only a single codec.

To do this, you will need to provide an empty mapping for content-type-to-codec to override the default:

    additional_codecs => { }

And to provide a single codec matching your expected content-type; for JSON this would be:

    codec => json

OR

    codec => json {
      # JSON codec directives inside these squiggles
    }

Use global pipeline.ecs_compatibility in logstash.yml

Without the pipeline context, these codecs still fall back to global settings.

[yaauie]

I believe this is fully-resolved with version 3.5.1 of the HTTP input plugin. Please feel free to reopen with additional details if this continues to be a problem after upgrading that plugin.