Plugin crashes Logstash at startup on all recent versions as of this afternoon

[mike-sol]

I had successfully deployed Logstash 7.17.0 (OSS, AMD64) to all my environments, and then when I got to deploying to production, Logstash would not start. I narrowed it down to this (critical!) plugin.

I can reproduce the startup error in a clean environment as follows:

> docker run --name logstash -it --entrypoint /bin/sh docker.elastic.co/logstash/logstash-oss:7.17.0-amd64

$ bash

logstash@248ad05ef6bb:~$ logstash-plugin install logstash-input-cloudwatch_logs
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Validating logstash-input-cloudwatch_logs
Resolving mixin dependencies
Updating mixin dependencies logstash-mixin-aws
Installing logstash-input-cloudwatch_logs
Installation successful

logstash@248ad05ef6bb:~$ exec /usr/local/bin/docker-entrypoint 
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2022-02-15T23:28:52,833][INFO ][logstash.runner          ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
[2022-02-15T23:28:52,841][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.17.0", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[2022-02-15T23:28:52,843][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dls.cgroup.cpuacct.path.override=/, -Dls.cgroup.cpu.path.override=/]
[2022-02-15T23:28:52,869][INFO ][logstash.settings        ] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[2022-02-15T23:28:52,960][INFO ][logstash.settings        ] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[2022-02-15T23:28:53,253][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"15fc8f37-3700-4ba9-adaa-5c8b16595bca", :path=>"/usr/share/logstash/data/uuid"}
[2022-02-15T23:28:53,332][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<ArgumentError: wrong number of arguments (given 3, expected 1..2)>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/api/modules/base.rb:43:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.2.0/lib/sinatra/base.rb:1537:in `new'", "/usr/share/logstash/logstash-core/lib/logstash/api/rack_app.rb:113:in `block in app'", "org/jruby/RubyBasicObject.java:2622:in `instance_eval'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-2.2.3/lib/rack/builder.rb:125:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/api/rack_app.rb:101:in `app'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:123:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:74:in `from_settings'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:76:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:520:in `create_agent'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:411:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:68:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:292:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:133:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:93:in `<main>'"]}
[2022-02-15T23:28:53,336][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
    at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
    at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
    at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]

Without this plugin installed Logstash starts fine. I can verify the same crash happens on 7.15 and 7.16 as well. It only started a few hours ago - my totally identical installations this morning went totally fine.

[mike-sol]

The strange thing is that I do not see a release of a new gem or any commits to this repo today....

[lisuml]

I'm facing similar issue. Did you find the solution?

[dizlajk]

https://github.com/elastic/logstash/issues/13777 should be fixed in Logstash 7.17.1 and 8.0.1