Bump Logstash Output S3 to 4.4

[bradfordlittooysonos]

The logstash output s3 plugin (https://github.com/logstash-plugins/logstash-output-s3/blob/main/CHANGELOG.md#440) recently moved to 4.4 and has a critical fix that handles corrupt files. I would like to use the latest version of this plugin, and have logstash-integration-aws use the latest version.

[mashhurs]

This issue was handled by #16 PR.

[bradfordlittooysonos]

I have filed a support ticket for this as well - but for visibility, on I am getting the following when using: docker.elastic.co/logstash/logstash:7.17.7 .

[2023-01-26T20:48:45,052][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:0_main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (RuntimeError) \n\n\tyou might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars withlock_jarscommand\n\nno such file to load -- org/logstash/plugins/integration/aws/logstash-integration-aws/7.1.0/logstash-integration-aws-7.1.0 (LoadError)", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:86)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:84)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:86)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:73)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:333)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:333)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:549)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:92)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:191)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:178)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:208)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:397)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:206)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:325)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:137)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:60)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:829)"]}

I am installing the following:

RUN logstash-plugin install --preserve logstash-input-dead_letter_queue logstash-codec-json_lines logstash-filter-prune logstash-input-lumberjack logstash-filter-memcached logstash-input-kinesis logstash-codec-cloudwatch_logs logstash-input-tcp

# Remove pre-installed plugins so the logstash-integration-aws plugin takes precedence 
RUN logstash-plugin remove logstash-output-s3
RUN logstash-plugin remove logstash-input-s3 
RUN logstash-plugin remove logstash-input-sqs
RUN logstash-plugin remove logstash-output-cloudwatch
RUN logstash-plugin remove logstash-output-sns
RUN logstash-plugin remove logstash-output-sqs

RUN logstash-plugin install --version 7.1.0 logstash-integration-aws
RUN logstash-plugin install --version 2.2.0.pre logstash-input-s3-sns-sqs

[yaauie]

@bradfordlittooysonos

I am not sure what the issue is, and this ticket isn't an appropriate place for a back-and-forth debugging about upgrading the plugin (please chase through your support rep, they will work with us to escalate/chase).

But: this issue is also fixed in v4.3.8 of the stand-alone s3 filter plugin (v4.4.1 of that plugin is latest and includes other fixes). The quickest path to get that on a fresh install of Logstash would be to only update logstash-output-s3, and to not install the aws integration plugin.