search

elastic / logstash

Logstash - transport and process your logs, events, or other data
https://www.elastic.co/products/logstash
Other
14.09k stars 3.48k forks source link

logstash-plugin list throws Gem::LoadError on list command #15687

Closed Kortekaasy closed 5 months ago

Kortekaasy commented 6 months ago

Logstash information:

Please include the following information:

  1. Logstash version (e.g. bin/logstash --version): logstash 8.11.3
  2. Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker): APT
  3. How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes): systemd

Plugins installed: (bin/logstash-plugin list --verbose): NA - this is the part that crashes

JVM (e.g. java -version): openjdk version "17.0.9" 2023-10-17 (Bundled)

OS version (uname -a if on a Unix-like system): Linux ip-10-81-4-10 6.1.0-15-cloud-arm64 #1 SMP Debian 6.1.66-1 (2023-12-09) aarch64 GNU/Linux

Description of the problem including expected versus actual behavior: After performing a fresh install of Logstash 8.11.3, I wanted to list the plugins using bin/logstash-plugin list. This gives the following output:

Using bundled JDK: /usr/share/logstash/jdk
io/console on JRuby shells out to stty for most operations
Gem::LoadError: You have already activated ffi 1.16.3, but your Gemfile requires ffi 1.15.5. Since ffi is a default gem, you can either remove your dependency on it or try updating to a newer version of bundler that supports ffi as a default gem.
  check_for_activated_spec! at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/runtime.rb:308
                      setup at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/runtime.rb:25
                       each at org/jruby/RubyArray.java:1989
                       each at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/spec_set.rb:155
                        map at org/jruby/RubyEnumerable.java:835
                      setup at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/runtime.rb:24
                      setup at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler.rb:161
                     setup! at /usr/share/logstash/lib/bootstrap/bundler.rb:83
                    execute at /usr/share/logstash/lib/pluginmanager/list.rb:33
                        run at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/clamp-1.0.1/lib/clamp/command.rb:68
                    execute at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/clamp-1.0.1/lib/clamp/subcommand/execution.rb:11
                        run at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/clamp-1.0.1/lib/clamp/command.rb:68
                        run at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/clamp-1.0.1/lib/clamp/command.rb:133
                     <main> at /usr/share/logstash/lib/pluginmanager/main.rb:64

On Logstash 8.11.2, the same command would give the output:

Using bundled JDK: /usr/share/logstash/jdk
io/console on JRuby shells out to stty for most operations
logstash-codec-avro
logstash-codec-cef
logstash-codec-collectd
logstash-codec-dots
logstash-codec-edn
logstash-codec-edn_lines
logstash-codec-es_bulk
logstash-codec-fluent
logstash-codec-graphite
logstash-codec-json
logstash-codec-json_lines
logstash-codec-line
logstash-codec-msgpack
logstash-codec-multiline
logstash-codec-netflow
logstash-codec-plain
logstash-codec-rubydebug
logstash-filter-aggregate
logstash-filter-anonymize
logstash-filter-cidr
logstash-filter-clone
logstash-filter-csv
logstash-filter-date
logstash-filter-de_dot
logstash-filter-dissect
logstash-filter-dns
logstash-filter-drop
logstash-filter-elasticsearch
logstash-filter-fingerprint
logstash-filter-geoip
logstash-filter-grok
logstash-filter-http
logstash-filter-json
logstash-filter-kv
logstash-filter-memcached
logstash-filter-metrics
logstash-filter-mutate
logstash-filter-prune
logstash-filter-ruby
logstash-filter-sleep
logstash-filter-split
logstash-filter-syslog_pri
logstash-filter-throttle
logstash-filter-translate
logstash-filter-truncate
logstash-filter-urldecode
logstash-filter-useragent
logstash-filter-uuid
logstash-filter-xml
logstash-input-azure_event_hubs
logstash-input-beats
└── logstash-input-elastic_agent (alias)
logstash-input-couchdb_changes
logstash-input-dead_letter_queue
logstash-input-elastic_serverless_forwarder
logstash-input-elasticsearch
logstash-input-exec
logstash-input-file
logstash-input-ganglia
logstash-input-gelf
logstash-input-generator
logstash-input-graphite
logstash-input-heartbeat
logstash-input-http
logstash-input-http_poller
logstash-input-imap
logstash-input-jms
logstash-input-pipe
logstash-input-redis
logstash-input-snmp
logstash-input-snmptrap
logstash-input-stdin
logstash-input-syslog
logstash-input-tcp
logstash-input-twitter
logstash-input-udp
logstash-input-unix
logstash-integration-aws
 ├── logstash-codec-cloudfront
 ├── logstash-codec-cloudtrail
 ├── logstash-input-cloudwatch
 ├── logstash-input-s3
 ├── logstash-input-sqs
 ├── logstash-output-cloudwatch
 ├── logstash-output-s3
 ├── logstash-output-sns
 └── logstash-output-sqs
logstash-integration-elastic_enterprise_search
 ├── logstash-output-elastic_app_search
 └──  logstash-output-elastic_workplace_search
logstash-integration-jdbc
 ├── logstash-input-jdbc
 ├── logstash-filter-jdbc_streaming
 └── logstash-filter-jdbc_static
logstash-integration-kafka
 ├── logstash-input-kafka
 └── logstash-output-kafka
logstash-integration-logstash
 ├── logstash-input-logstash
 └── logstash-output-logstash
logstash-integration-rabbitmq
 ├── logstash-input-rabbitmq
 └── logstash-output-rabbitmq
logstash-output-csv
logstash-output-elasticsearch
logstash-output-email
logstash-output-file
logstash-output-graphite
logstash-output-http
logstash-output-lumberjack
logstash-output-nagios
logstash-output-null
logstash-output-pipe
logstash-output-redis
logstash-output-stdout
logstash-output-tcp
logstash-output-udp
logstash-output-webhdfs
logstash-patterns-core

Steps to reproduce:

  1. Take a fresh Debian 12 machine. In my case I created an instance in AWS with the latest Debian 12 AMI.
  2. Install logstash 8.11.3 using the instructions provided at https://www.elastic.co/guide/en/logstash/8.11/installing-logstash.html#_apt
  3. Run /usr/share/logstash/bin/logstash-plugin list

Provide logs (if relevant): Logs (as shown above):

Using bundled JDK: /usr/share/logstash/jdk
io/console on JRuby shells out to stty for most operations
Gem::LoadError: You have already activated ffi 1.16.3, but your Gemfile requires ffi 1.15.5. Since ffi is a default gem, you can either remove your dependency on it or try updating to a newer version of bundler that supports ffi as a default gem.
  check_for_activated_spec! at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/runtime.rb:308
                      setup at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/runtime.rb:25
                       each at org/jruby/RubyArray.java:1989
                       each at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/spec_set.rb:155
                        map at org/jruby/RubyEnumerable.java:835
                      setup at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/runtime.rb:24
                      setup at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler.rb:161
                     setup! at /usr/share/logstash/lib/bootstrap/bundler.rb:83
                    execute at /usr/share/logstash/lib/pluginmanager/list.rb:33
                        run at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/clamp-1.0.1/lib/clamp/command.rb:68
                    execute at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/clamp-1.0.1/lib/clamp/subcommand/execution.rb:11
                        run at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/clamp-1.0.1/lib/clamp/command.rb:68
                        run at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/clamp-1.0.1/lib/clamp/command.rb:133
                     <main> at /usr/share/logstash/lib/pluginmanager/main.rb:64
chrisdudley commented 6 months ago

I am seeing the same issue with both the MacOSX-arm and Linux/x86 variants of 8.11.3. 8.11.2 worked fine. Both bin/logstash-plugin list and bin/logstash-plugin remove return errors.

mdetrano commented 6 months ago

Seeing the same Gem dependency (for ffi) error trying: prepare-offline-pack.

BrandonHigbee commented 6 months ago

Is there an easy way to target an older version using similar installation instructions as the docs?

elster commented 6 months ago

Is there an easy way to target an older version using similar installation instructions as the docs?

I would also be interested, especially for yum updates.

elster commented 6 months ago

Is there an easy way to target an older version using similar installation instructions as the docs?

I would also be interested, especially for yum updates.

dnf install logstash --version 1:8.11.3-1

jsvd commented 6 months ago

Is there an easy way to target an older version using similar installation instructions as the docs?

Package managers and docker have ways to specify which version to install/run:

yum --showduplicate list logstash
yum install logstash-<version.architecture>
apt-get install logstash=<version>
docker run -it logstash:version -e "input { http { port => 3333 }}"

As for the bug, the fix https://github.com/elastic/logstash/pull/15694 has been merged to 8.11, thanks @robbavey. The upcoming 8.12.0 release won't have this issue, which is targeted for the first half of January. I'll have to see if we can get a 8.11.4 out, but may be difficult with the holidays.

BrandonHigbee commented 6 months ago

For others that might struggle to find what version to specify using apt-get here is an example configuration:

RUN apt-get update -y && \
    apt-get install -y wget gnupg apt-transport-https && \
    wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add - && \
    echo "deb https://artifacts.elastic.co/packages/oss-8.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-oss-8.x.list && \
    apt-get update -y && \
    apt-get install -y logstash-oss=1:8.11.2-1

The only change to specify the version is on the last line, it took me a while to figure out the version is 1:8.11.2-1.

C0rn3j commented 5 months ago

Fix merged and 8.12 is out as of two days ago and 8.11.4 about a week or so, can be closed I suppose. (I haven't tested it)

chrisdudley commented 5 months ago

I can confirm this problem is fixed in 8.11.4 and 8.12.

Kortekaasy commented 5 months ago

Issue seems fixed on my end as well. Closing it seems appropriate