Please update to AWS SDK v3

st33v commented 3 years ago

We're trying to use this plugin along with AWS EKS service account roles and it does not seem to work because while 'logstash-input-s3' has been updated to AWS SDK v3 here, this change does not seem to have made it into the 'logstash-output-s3' plugin.

iamcan13 commented 3 years ago

Could you please help with this? We also get pain from this.

WilliamDuTrendMicro commented 2 years ago

yes, it is required to run logstash on EKS with S3 output

jsvd commented 2 years ago

you can install a new integration plugin that moves all aws-related plugins to the new sdk. However you need to uninstall all individual plugins before installing logstash-integration-aws:

/tmp/logstash-8.3.2
❯ for plugin in logstash-codec-cloudfront logstash-codec-cloudtrail logstash-input-cloudwatch logstash-input-s3 logstash-input-sqs logstash-output-cloudwatch logstash-output-sns logstash-output-sqs logstash-output-s3; do bin/logstash-plugin remove $plugin;done
Using bundled JDK: /tmp/logstash-8.3.2/jdk.app/Contents/Home
ERROR: Operation aborted, cannot remove plugin, message: This plugin has not been previously installed
Using bundled JDK: /tmp/logstash-8.3.2/jdk.app/Contents/Home
ERROR: Operation aborted, cannot remove plugin, message: This plugin has not been previously installed
Using bundled JDK: /tmp/logstash-8.3.2/jdk.app/Contents/Home
ERROR: Operation aborted, cannot remove plugin, message: This plugin has not been previously installed
Using bundled JDK: /tmp/logstash-8.3.2/jdk.app/Contents/Home
Resolving dependencies........................................................................................................................................................................................................................................................
Resolving dependencies...............................................................
Successfully removed logstash-input-s3
Using bundled JDK: /tmp/logstash-8.3.2/jdk.app/Contents/Home
Resolving dependencies......................................................................................................................................................................................................................................................
Resolving dependencies...................................................................................................
Successfully removed logstash-input-sqs
Using bundled JDK: /tmp/logstash-8.3.2/jdk.app/Contents/Home
Resolving dependencies.....................................................................................................................................................................................................................................................
Resolving dependencies...............................................................
Successfully removed logstash-output-cloudwatch
Using bundled JDK: /tmp/logstash-8.3.2/jdk.app/Contents/Home
Resolving dependencies....................................................................................................................................................................................................................................................
Resolving dependencies.......................................................................
Successfully removed logstash-output-sns
Using bundled JDK: /tmp/logstash-8.3.2/jdk.app/Contents/Home
Resolving dependencies..................................................................................................................................................................................................................................................
Resolving dependencies..................................................................................................
Successfully removed logstash-output-sqs
Using bundled JDK: /tmp/logstash-8.3.2/jdk.app/Contents/Home
Resolving dependencies.................................................................................................................................................................................................................................................
Resolving dependencies.............................................................
Successfully removed logstash-output-s3

/tmp/logstash-8.3.2 3m 41s
❯ bin/logstash-plugin install logstash-integration-aws                   
Using bundled JDK: /tmp/logstash-8.3.2/jdk.app/Contents/Home
Validating logstash-integration-aws
Resolving mixin dependencies
Installing logstash-integration-aws
Installation successful

/tmp/logstash-8.3.2
❯ grep "^    aws-sdk-core" Gemfile.lock
    aws-sdk-core (3.131.4)

WilliamDuTrendMicro commented 2 years ago

works in EKS with IRSA

RUN sudo wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
RUN sudo apt-get install apt-transport-https
RUN sudo echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

RUN sudo apt-get update && sudo apt-get install logstash=1:7.4.2-1 -y

#RUN /usr/share/logstash/bin/logstash-plugin install logstash-output-amazon_es
RUN sudo apt install jq cron less -y
RUN cd /usr/share/logstash/ \
    && for plugin in logstash-codec-cloudfront logstash-codec-cloudtrail logstash-input-cloudwatch logstash-input-s3 logstash-input-sqs logstash-output-cloudwatch logstash-output-sns logstash-output-sqs logstash-output-s3; do  /usr/share/logstash/bin/logstash-plugin remove $plugin;done \
    && /usr/share/logstash/bin/logstash-plugin install logstash-integration-aws 
#https://github.com/logstash-plugins/logstash-output-s3/issues/234#issuecomment-1197902180
#as the S3 output does not support IRSA due to aws sdk low verison , enforce unintall and install logstash-integration-aws with AWS SDK latest version.

logstash-plugins / logstash-output-s3

Please update to AWS SDK v3 #234