search

opensearch-project / logstash-output-opensearch

A Logstash plugin that sends event data to a OpenSearch clusters and stores as an index.
https://opensearch.org/docs/latest/clients/logstash/index/
Apache License 2.0
104 stars 80 forks source link

[BUG] Unable to create logstash image with logstash-output-opensearch plugin #209

Closed FloMedja closed 5 months ago

FloMedja commented 1 year ago

Describe the bug

Unable to install logstash-opensearch-plugin in a logstash-oss docker image

To Reproduce

  1. Create a Dockerfile with the following content
ARG APP_VERSION

FROM docker.elastic.co/logstash/logstash-oss:${APP_VERSION}

# Remove existing logstash aws plugins and install logstash-integration-aws to keep sdk dependency the same
# https://github.com/logstash-plugins/logstash-mixin-aws/issues/38
# https://github.com/opensearch-project/logstash-output-opensearch#configuration-for-logstash-output-opensearch-plugin
RUN logstash-plugin remove logstash-input-s3
RUN logstash-plugin remove logstash-input-sqs
RUN logstash-plugin remove logstash-output-s3
RUN logstash-plugin remove logstash-output-sns
RUN logstash-plugin remove logstash-output-sqs
RUN logstash-plugin remove logstash-output-cloudwatch

RUN logstash-plugin install --version 7.1.1 logstash-integration-aws

RUN logstash-plugin install --version 2.0.1 logstash-output-opensearch

  1. Run docker build --build-arg APP_VERSION=7.17.10 .

  2. The image creation fails with the following error

Step 10/10 : RUN logstash-plugin install --version 2.0.1 logstash-output-opensearch
 ---> Running in 4b6616f16499
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Validating logstash-output-opensearch-2.0.1
Resolving mixin dependencies
Updating mixin dependencies logstash-mixin-ecs_compatibility_support
Bundler attempted to update logstash-mixin-ecs_compatibility_support but its version stayed the same
Installing logstash-output-opensearch
Plugin version conflict, aborting
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "json":
  In snapshot (Gemfile.lock):
    json (= 1.8.6)

  In Gemfile:
    json (~> 1)

    logstash-output-opensearch (= 2.0.1) was resolved to 2.0.1, which depends on
      json (>= 2.3.0, ~> 2)

Deleting your Gemfile.lock file and running `bundle install` will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.
The command '/bin/sh -c logstash-plugin install --version 2.0.1 logstash-output-opensearch' returned a non-zero code: 1

Expected behavior

The logstash image is created with the logstash-output-opensearch plugin

Additional context

I also have the same errors with the version 2.0.0 and 1.3.0 of logstash-output-opensearch.

oeyh commented 1 year ago

@FloMedja Thanks for raising the issue! Looks like there's a dependency conflict.

If you are OK to use Logstash 8.x, the following should work if APP_VERSION is 8.x, for example 8.7.0, 8.5.3, etc.

ARG APP_VERSION

FROM docker.elastic.co/logstash/logstash-oss:${APP_VERSION}
RUN logstash-plugin install --version 7.1.1 logstash-integration-aws
RUN logstash-plugin install --version 2.0.1 logstash-output-opensearch

If you need to use Logstash 7.x, this might be a workaround: Clone the repo, remove this line that adds the json version spec and build your own gem with:

gem build logstash-output-opensearch.gemspec

This should generate a gem, e.g., logstash-output-opensearch-2.0.1.gem Then in the Dockerfile, you can copy and install the gem. Something like this might work:

ARG APP_VERSION

FROM docker.elastic.co/logstash/logstash-oss:${APP_VERSION}
# Remove existing logstash aws plugins and install logstash-integration-aws to keep sdk dependency the same
# https://github.com/logstash-plugins/logstash-mixin-aws/issues/38
# https://github.com/opensearch-project/logstash-output-opensearch#configuration-for-logstash-output-opensearch-plugin
RUN logstash-plugin remove logstash-input-s3
RUN logstash-plugin remove logstash-input-sqs
RUN logstash-plugin remove logstash-output-s3
RUN logstash-plugin remove logstash-output-sns
RUN logstash-plugin remove logstash-output-sqs
RUN logstash-plugin remove logstash-output-cloudwatch

RUN logstash-plugin install --version 7.1.1 logstash-integration-aws

COPY path/to/local/gem/logstash-output-opensearch-2.0.1.gem /usr/share
RUN logstash-plugin install /usr/share/logstash-output-opensearch-2.0.1.gem
FloMedja commented 1 year ago

Thanks @oeyh . I was able to install the plugin in my docker image. But when try to use the opensearch plugin in the created image I got this error

[2023-06-14T22:15:13,101][ERROR][logstash.plugins.registry] Unable to load plugin. {:type=>"output", :name=>"opensearch"}
[2023-06-14T22:15:13,111][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
 at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
 at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]

My output definition looks like this : 

output {

  opensearch {
    hosts => [\"https://my-host.es.amazonaws.com/\"]
    auth_type => {
      type => 'aws_iam'
      region => \"us-east-1\"
      aws_access_key_id => \"user\"
      aws_secret_access_key => \"pass"
    }
    index => \"%{[@metadata][index_name]}\"
  }
}

When I execute a logstash-plugin list in the logstash container I have the following output:

Using bundled JDK: /usr/share/logstash/jdk
logstash-codec-avro
logstash-codec-cef
logstash-codec-collectd
logstash-codec-dots
logstash-codec-edn
logstash-codec-edn_lines
logstash-codec-es_bulk
logstash-codec-fluent
logstash-codec-graphite
logstash-codec-json
logstash-codec-json_lines
logstash-codec-line
logstash-codec-msgpack
logstash-codec-multiline
logstash-codec-netflow
logstash-codec-plain
logstash-codec-rubydebug
logstash-filter-aggregate
logstash-filter-anonymize
logstash-filter-cidr
logstash-filter-clone
logstash-filter-csv
logstash-filter-date
logstash-filter-de_dot
logstash-filter-dissect
logstash-filter-dns
logstash-filter-drop
logstash-filter-elasticsearch
logstash-filter-fingerprint
logstash-filter-geoip
logstash-filter-grok
logstash-filter-http
logstash-filter-json
logstash-filter-kv
logstash-filter-memcached
logstash-filter-metrics
logstash-filter-mutate
logstash-filter-prune
logstash-filter-ruby
logstash-filter-sleep
logstash-filter-split
logstash-filter-syslog_pri
logstash-filter-throttle
logstash-filter-translate
logstash-filter-truncate
logstash-filter-urldecode
logstash-filter-useragent
logstash-filter-uuid
logstash-filter-xml
logstash-input-azure_event_hubs
logstash-input-beats
└── logstash-input-elastic_agent (alias)
logstash-input-couchdb_changes
logstash-input-dead_letter_queue
logstash-input-elasticsearch
logstash-input-exec
logstash-input-file
logstash-input-ganglia
logstash-input-gelf
logstash-input-generator
logstash-input-graphite
logstash-input-heartbeat
logstash-input-http
logstash-input-http_poller
logstash-input-imap
logstash-input-jms
logstash-input-pipe
logstash-input-redis
logstash-input-snmp
logstash-input-snmptrap
logstash-input-stdin
logstash-input-syslog
logstash-input-tcp
logstash-input-twitter
logstash-input-udp
logstash-input-unix
logstash-integration-aws
 ├── logstash-codec-cloudfront
 ├── logstash-codec-cloudtrail
 ├── logstash-input-cloudwatch
 ├── logstash-input-s3
 ├── logstash-input-sqs
 ├── logstash-output-cloudwatch
 ├── logstash-output-s3
 ├── logstash-output-sns
 └── logstash-output-sqs
logstash-integration-elastic_enterprise_search
 ├── logstash-output-elastic_app_search
 └──  logstash-output-elastic_workplace_search
logstash-integration-jdbc
 ├── logstash-input-jdbc
 ├── logstash-filter-jdbc_streaming
 └── logstash-filter-jdbc_static
logstash-integration-kafka
 ├── logstash-input-kafka
 └── logstash-output-kafka
logstash-integration-rabbitmq
 ├── logstash-input-rabbitmq
 └── logstash-output-rabbitmq
logstash-output-csv
logstash-output-elasticsearch
logstash-output-email
logstash-output-file
logstash-output-graphite
logstash-output-http
logstash-output-lumberjack
logstash-output-nagios
logstash-output-null
logstash-output-opensearch
logstash-output-pipe
logstash-output-redis
logstash-output-stdout
logstash-output-tcp
logstash-output-udp
logstash-output-webhdfs
logstash-patterns-core

Additional context

I have logstash version 7.17.10 and opensearch plugin v2.0.1.

FloMedja commented 1 year ago

@oeyh any update on this ?

FloMedja commented 1 year ago

Up !

oeyh commented 1 year ago

@FloMedja Sorry for the late reply. Great to see it working! I'm not familiar with the error you saw. What was the issue?

FloMedja commented 1 year ago

@oeyh sorry I missed your reply.

Even when I installed the opensearch-output-plugin in the image, It can't be load when I have a pipeline using the plugin. I have the following error saying that the plugin can't be loaded. without no much details. 

[2023-06-14T22:15:13,101][ERROR][logstash.plugins.registry] Unable to load plugin. {:type=>"output", :name=>"opensearch"}
[2023-06-14T22:15:13,111][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
 at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
 at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]

I tried to install the plugin from different folder in logstash image without success. Each time the command logstash-plugin list shows the opensearch-output-plugin installed but it can't be loaded.

In your opensearchproject/logstash-oss-with-opensearch-output-plugin for the tag 7.16.3 how do you make the plugin installation ? Can you share your Dockerfile please ?

oeyh commented 1 year ago

This Dockerfile worked for me. I was able to use the opensearch plugin to write to OpenSearch service.

APP_VERSION was 7.17.10 and logstash-output-opensearch-2.0.1-x86_64-linux.gem was built following the steps in a previous comment.

ARG APP_VERSION

FROM docker.elastic.co/logstash/logstash-oss:${APP_VERSION}
# Remove existing logstash aws plugins and install logstash-integration-aws to keep sdk dependency the same
# https://github.com/logstash-plugins/logstash-mixin-aws/issues/38
# https://github.com/opensearch-project/logstash-output-opensearch#configuration-for-logstash-output-opensearch-plugin
RUN logstash-plugin remove logstash-input-s3
RUN logstash-plugin remove logstash-input-sqs
RUN logstash-plugin remove logstash-output-s3
RUN logstash-plugin remove logstash-output-sns
RUN logstash-plugin remove logstash-output-sqs
RUN logstash-plugin remove logstash-output-cloudwatch

RUN logstash-plugin install --version 7.1.1 logstash-integration-aws

COPY ./logstash-output-opensearch-2.0.1-x86_64-linux.gem /usr/share
RUN logstash-plugin install /usr/share/logstash-output-opensearch-2.0.1-x86_64-linux.gem
dblock commented 11 months ago

This was helpful, I needed to make an image and did the following.

To make a local image without a custom dockerfile I was able to modify release/docker/build-docker-multi-arch.sh as follows (remove --push, remove multi-arch leaving just one platform, add --output type=docker).

-docker buildx build --platform linux/amd64,linux/arm64 --build-arg VERSION=$VERSION -t $REPOSITORY_NAME/logstash-oss-with-opensearch-output-plugin:$VERSION -t $REPOSITORY_NAME/logstash-oss-with-opensearch-output-plugin:latest -f $DOCKER_FILE_PATH --push .
+docker buildx build --output type=docker --platform linux/amd64 --build-arg VERSION=$VERSION -t $REPOSITORY_NAME/logstash-oss-with-opensearch-output-plugin:$VERSION -t $REPOSITORY_NAME/logstash-oss-with-opensearch-output-plugin:latest -f $DOCKER_FILE_PATH .
$ cd release/docker

$ ./build-docker-multi-arch.sh -v 7.17.12 -t remote -r opensearchproject
 => [1/3] FROM docker.elastic.co/logstash/logstash-oss:7.17.12@sha256:1260443ab01ade254174232a1194e082eb2be50f6f8b502a583c2466254a696f                                                5.9s
...
 => importing to docker        
$ docker image ls | grep opensearchproject

opensearchproject/logstash-oss-with-opensearch-output-plugin                          7.17.12                        d4b2459705a4   45 seconds ago   829MB
$ docker run --rm -it --entrypoint bash opensearchproject/logstash-oss-with-opensearch-output-plugin
$ ./bin/logstash-plugin list --verbose | grep logstash-output-opensearch

logstash-output-opensearch (1.3.0)

Let's upgrade.

$ ./bin/logstash-plugin remove logstash-input-s3
$ ./bin/logstash-plugin remove logstash-input-sqs
$ ./bin/logstash-plugin remove logstash-output-s3
$ ./bin/logstash-plugin remove logstash-output-sns
$ ./bin/logstash-plugin remove logstash-output-sqs
$ ./bin/logstash-plugin remove logstash-output-cloudwatch
$ ./bin/logstash-plugin remove logstash-output-opensearch

$ ./bin/logstash-plugin install --version 7.1.1 logstash-integration-aws
$ ./bin/logstash-plugin install --version 2.0.2 logstash-output-opensearch

$ ./bin/logstash-plugin list --verbose | grep logstash-output-opensearch
logstash-output-opensearch (2.0.2)
FloMedja commented 5 months ago

@oeyh Just an update on this. I was able to make it work. I was doing a mistake in the gem building. Thanks for the support.

oeyh commented 5 months ago

@FloMedja Glad it worked for you. Resolving the issue.

dblock commented 5 months ago

@oeyh Just an update on this. I was able to make it work. I was doing a mistake in the gem building. Thanks for the support.

Want to contribute a guide/documentation to this repo along the lines of what that takes?

FloMedja commented 5 months ago

@dblock for sure. @oeyh In which session of the documentation should I put the instructions ?

oeyh commented 5 months ago

@FloMedja Thanks! I think we can add a doc under /docs and link it in README.md.

FloMedja commented 5 months ago

@oeyh I create a pull request