Support SQS message user attributes

piequi commented 1 year ago

Release notes

Added support for SQS message user attributes

What does this PR do?

This PR adds the ability to the SQS input plugin to also read and decode SQS user attributes that may be contained in the message.

When some user attributes are defined, the message_attributes key contains a hash of String -> Aws::SQS::Types::MessageAttributeValue with the attributes' name and value.

https://docs.aws.amazon.com/zh_tw/sdk-for-ruby/v3/api/Aws/SQS/Types/MessageAttributeValue.html https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_MessageAttributeValue.html

Why is it important/What is the impact to the user?

Supporting user attributes will allow to enrich events with context metadata when the source emitting the SQS message uses a standard format that cannot be altered.

As mentioned in https://github.com/logstash-plugins/logstash-integration-aws/issues/22, a Python logging handler does not allow to add cloud metadata to a log record. This could only be achieved by adding user attributes when emitting the log record in a SQS message.

Checklist

[x] My code follows the style guidelines of this project
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] I have made corresponding change to the default configuration files (and/or docker env variables)
[ ] I have added tests that prove my fix is effective or that my feature works

Author's Checklist

[ ]

How to test this PR locally

Related issues

Closes #22

Use cases

Screenshots

Logs

cla-checker-service[bot] commented 1 year ago

💚 CLA has been signed

piequi commented 1 year ago

Applying instructions to run tests from https://www.elastic.co/guide/en/logstash/current/contributing-patch-plugin.html simply do not work.

bundle install fails with

Could not find gem 'logstash-devutils' with platform 'x86_64-linux' in rubygems repository https://rubygems.org/ or installed locally.

The source contains the following gems matching 'logstash-devutils':
  * logstash-devutils-0.0.1-java
  * logstash-devutils-0.0.2-java
[...]
  * logstash-devutils-2.4.0-java

Using jruby -S bundle install works

[...]
Using logstash-integration-aws 7.1.1 from source at `.`
Bundle complete! 8 Gemfile dependencies, 82 gems now installed.

but then jruby -S bundle exec rspec fails with

You have requested:
  logstash-devutils >= 0

The bundle currently has logstash-devutils locked at 1.3.6.
Try running `bundle update logstash-devutils`

If you are updating multiple gems in your Gemfile at once,
try passing them all to `bundle update`
Run `bundle install` to install missing gems.

Setting LOGSTASH_PATH environment variable do not change anything; got a hint from https://github.com/logstash-plugins/logstash-integration-aws/blob/183d113484b8a555abca91db00b2a6844274a258/Gemfile#L5

Looking around I discovered some gradle files and run gradlew assemble && gradlew test. Output is fine but I've no clue if this is running the correct unit tests...

Looking at Travis CI, running jruby -S rake test seems to be the way to go but logstash-devutils is definitely missing...

@logstash-core some help would be much appreciated !

jsvd commented 1 year ago

@piequi first of all many thanks for this contribution!!

So here's a set of steps that should get you going:

Step 1: install jruby 9.3 locally

curl https://repo1.maven.org/maven2/org/jruby/jruby-dist/9.3.10.0/jruby-dist-9.3.10.0-bin.tar.gz | tar -zxf - -C /tmp

Step 2: install Logstash

tar -zxf ~/Downloads/logstash-8.6.2-darwin-aarch64.tar.gz -C /tmp

Step 3: setup environment variables

export LOGSTASH_PATH=/tmp/logstash-8.6.2 LOGSTASH_SOURCE=1

Step 4: install bundler, dependencies and vendor jars

/tmp/jruby-9.3.10.0/bin/jruby -S gem install bundler 
/tmp/jruby-9.3.10.0/bin/jruby -S bundle install
/tmp/jruby-9.3.10.0/bin/jruby -S bundle exec rake vendor

Step 5: run tests

/tmp/jruby-9.3.10.0/bin/jruby -S bundle exec rspec

Those docs are in serious need of updating, I'll track this need and hopefully get to it soon.

piequi commented 1 year ago

Hi @jsvd ! Thanks for your help !

Following your instructions, everything goes fine until jruby -S bundle exec rake vendor where I get:

[!] There was an error while loading `logstash-core-plugin-api.gemspec`: The logstash-core-api need to be build on jruby. Bundler cannot continue.

 #  from /home/marc-antoine/bin/logstash-8.6.2/logstash-core-plugin-api/logstash-core-plugin-api.gemspec:46
 #  -------------------------------------------
 #    else
 >      raise "The logstash-core-api need to be build on jruby"
 #    end
 #  -------------------------------------------

It seems some commands aren't executed using jruby as I see that :

$ irb
irb(main):001:0> puts RUBY_PLATFORM
x86_64-linux-gnu
=> nil

$ jruby -S irb                    
irb(main):001:0> puts RUBY_PLATFORM
java
=> nil

For info:

$ jruby -v    
jruby 9.3.10.0 (2.6.8) 2023-02-01 107b2e6697 OpenJDK 64-Bit Server VM 11.0.17+8-post-Ubuntu-1ubuntu2 on 11.0.17+8-post-Ubuntu-1ubuntu2 +jit [x86_64-linux]

$ ruby -v                          
ruby 3.0.4p208 (2022-04-12 revision 3fa771dded) [x86_64-linux-gnu]

(Running the exact commands you provided and extracting JRuby to /tmp gives the same ouputs)

piequi commented 1 year ago

Here is a Dockerfile highlighting the error I get :

FROM ubuntu:latest

USER root

RUN apt-get update
RUN apt-get install -y curl default-jre git rake

RUN adduser --disabled-password  test

USER test
WORKDIR /home/test

RUN curl https://repo1.maven.org/maven2/org/jruby/jruby-dist/9.3.10.0/jruby-dist-9.3.10.0-bin.tar.gz \
  | tar -zxf -

RUN curl https://artifacts.elastic.co/downloads/logstash/logstash-oss-8.6.2-linux-x86_64.tar.gz \
  | tar -zxf -

ENV LOGSTASH_PATH=/home/test/logstash-8.6.2
ENV LOGSTASH_SOURCE=1

RUN /home/test/jruby-9.3.10.0/bin/jruby -S gem install bundler rspec

RUN git clone https://github.com/logstash-plugins/logstash-integration-aws.git

WORKDIR /home/test/logstash-integration-aws

RUN /home/test/jruby-9.3.10.0/bin/jruby -S bundle install
RUN /home/test/jruby-9.3.10.0/bin/jruby -S bundle exec rake vendor
RUN /home/test/jruby-9.3.10.0/bin/jruby -S bundle exec rspec

jsvd commented 1 year ago

Here are a few tweaks:

FROM ubuntu:latest

USER root

RUN apt-get update
RUN apt-get install -y curl default-jre git rake

RUN adduser --disabled-password  test

USER test
WORKDIR /home/test

RUN curl https://repo1.maven.org/maven2/org/jruby/jruby-dist/9.3.10.0/jruby-dist-9.3.10.0-bin.tar.gz \
  | tar -zxf -

ENV PATH="/home/test/jruby-9.3.10.0/bin/:${PATH}"

RUN curl https://artifacts.elastic.co/downloads/logstash/logstash-8.6.2-linux-x86_64.tar.gz \
  | tar -zxf -

ENV LOGSTASH_PATH=/home/test/logstash-8.6.2
ENV LOGSTASH_SOURCE=1

RUN jruby -S gem install bundler rspec

RUN git clone https://github.com/logstash-plugins/logstash-integration-aws.git

WORKDIR /home/test/logstash-integration-aws

RUN bundle install
RUN bundle exec rake vendor
RUN bundle exec rspec

piequi commented 1 year ago

Thanks @jsvd ! I'm good to fix existing tests and add a few more to validate my changes !

logstash-plugins / logstash-integration-aws