elastic / logstash

Logstash - transport and process your logs, events, or other data
https://www.elastic.co/products/logstash
Other
14.18k stars 3.5k forks source link

./plugin pack - missing pack plugins #4523

Closed robgil closed 7 years ago

robgil commented 8 years ago

logstash 2.1.1-1 centos6

When creating a pack, some plugins are not included. The ones I have found to be missing are logstash-output-webhdfs and logstash-output-mongo. These both have java in common.

List of plugins from host that was used to create the pack

The plugins on this host were installed via the typical ./plugin install

# ./plugin list
logstash-codec-collectd
logstash-codec-dots
logstash-codec-edn
logstash-codec-edn_lines
logstash-codec-es_bulk
logstash-codec-fluent
logstash-codec-graphite
logstash-codec-json
logstash-codec-json_lines
logstash-codec-line
logstash-codec-msgpack
logstash-codec-multiline
logstash-codec-netflow
logstash-codec-oldlogstashjson
logstash-codec-plain
logstash-codec-rubydebug
logstash-filter-anonymize
logstash-filter-checksum
logstash-filter-clone
logstash-filter-csv
logstash-filter-date
logstash-filter-dns
logstash-filter-drop
logstash-filter-fingerprint
logstash-filter-geoip
logstash-filter-grok
logstash-filter-json
logstash-filter-kv
logstash-filter-metrics
logstash-filter-multiline
logstash-filter-mutate
logstash-filter-ruby
logstash-filter-sleep
logstash-filter-split
logstash-filter-syslog_pri
logstash-filter-throttle
logstash-filter-urldecode
logstash-filter-useragent
logstash-filter-uuid
logstash-filter-xml
logstash-input-beats
logstash-input-couchdb_changes
logstash-input-elasticsearch
logstash-input-eventlog
logstash-input-exec
logstash-input-file
logstash-input-ganglia
logstash-input-gelf
logstash-input-generator
logstash-input-graphite
logstash-input-heartbeat
logstash-input-http
logstash-input-imap
logstash-input-irc
logstash-input-jdbc
logstash-input-kafka
logstash-input-log4j
logstash-input-lumberjack
logstash-input-pipe
logstash-input-rabbitmq
logstash-input-redis
logstash-input-s3
logstash-input-snmptrap
logstash-input-sqs
logstash-input-stdin
logstash-input-syslog
logstash-input-tcp
logstash-input-twitter
logstash-input-udp
logstash-input-unix
logstash-input-xmpp
logstash-input-zeromq
logstash-output-cloudwatch
logstash-output-csv
logstash-output-elasticsearch
logstash-output-email
logstash-output-exec
logstash-output-file
logstash-output-ganglia
logstash-output-gelf
logstash-output-graphite
logstash-output-hipchat
logstash-output-http
logstash-output-irc
logstash-output-juggernaut
logstash-output-kafka
logstash-output-lumberjack
logstash-output-mongodb
logstash-output-nagios
logstash-output-nagios_nsca
logstash-output-null
logstash-output-opentsdb
logstash-output-pagerduty
logstash-output-pipe
logstash-output-rabbitmq
logstash-output-redis
logstash-output-s3
logstash-output-sns
logstash-output-sqs
logstash-output-statsd
logstash-output-stdout
logstash-output-tcp
logstash-output-udp
logstash-output-webhdfs
logstash-output-xmpp
logstash-output-zeromq
logstash-patterns-core
# ./plugin pack
Packaging plugins for offline usage
Generated at /opt/logstash/plugins_package.tar.gz

List of gems from pack file in /opt/logstash/vendor/cache/ after unpack

cache # ls -1
addressable-2.3.8.gem
arr-pm-0.0.10.gem
atomic-1.1.99-java.gem
avl_tree-1.2.1.gem
awesome_print-1.6.1.gem
aws-sdk-2.1.36.gem
aws-sdk-core-2.1.36.gem
aws-sdk-resources-2.1.36.gem
aws-sdk-v1-1.66.0.gem
backports-3.6.7.gem
bindata-2.1.0.gem
bson-3.2.6-java.gem
buftok-0.2.0.gem
cabin-0.7.2.gem
childprocess-0.5.8.gem
cinch-2.3.1.gem
clamp-0.6.5.gem
coderay-1.1.0.gem
concurrent-ruby-0.9.2-java.gem
domain_name-0.5.25.gem
edn-1.1.0.gem
elasticsearch-1.0.15.gem
elasticsearch-api-1.0.15.gem
elasticsearch-transport-1.0.15.gem
equalizer-0.0.10.gem
faraday-0.9.2.gem
ffi-1.9.10-java.gem
ffi-rzmq-2.0.4.gem
ffi-rzmq-core-1.0.4.gem
file-dependencies-0.1.6.gem
filesize-0.0.4.gem
filewatch-0.6.7.gem
fpm-1.3.3.gem
gelf-1.3.2.gem
gelfd-0.2.0.gem
gems-0.8.3.gem
geoip-1.6.1.gem
gmetric-0.1.3.gem
hipchat-1.5.2.gem
hitimes-1.2.3-java.gem
http-0.9.8.gem
httparty-0.13.7.gem
http-cookie-1.0.2.gem
http-form_data-1.0.1.gem
http_parser.rb-0.6.0-java.gem
i18n-0.6.9.gem
jar-dependencies-0.3.1.gem
jls-grok-0.11.2.gem
jls-lumberjack-0.0.26.gem
jmespath-1.1.3.gem
jrjackson-0.3.7.gem
jruby-kafka-1.4.0-java.gem
jruby-openssl-0.9.12-java.gem
json-1.8.3-java.gem
logstash-codec-collectd-2.0.2.gem
logstash-codec-dots-2.0.2.gem
logstash-codec-edn-2.0.2.gem
logstash-codec-edn_lines-2.0.2.gem
logstash-codec-es_bulk-2.0.2.gem
logstash-codec-fluent-2.0.2-java.gem
logstash-codec-graphite-2.0.2.gem
logstash-codec-json-2.0.4.gem
logstash-codec-json_lines-2.0.2.gem
logstash-codec-line-2.0.2.gem
logstash-codec-msgpack-2.0.2-java.gem
logstash-codec-multiline-2.0.4.gem
logstash-codec-netflow-2.0.2.gem
logstash-codec-oldlogstashjson-2.0.2.gem
logstash-codec-plain-2.0.2.gem
logstash-codec-rubydebug-2.0.4.gem
logstash-core-2.1.1-java.gem
logstash-filter-anonymize-2.0.2.gem
logstash-filter-checksum-2.0.2.gem
logstash-filter-clone-2.0.4.gem
logstash-filter-csv-2.1.0.gem
logstash-filter-date-2.0.2.gem
logstash-filter-dns-2.0.2.gem
logstash-filter-drop-2.0.2.gem
logstash-filter-fingerprint-2.0.2.gem
logstash-filter-geoip-2.0.4.gem
logstash-filter-grok-2.0.2.gem
logstash-filter-json-2.0.2.gem
logstash-filter-kv-2.0.2.gem
logstash-filter-metrics-3.0.0.gem
logstash-filter-multiline-2.0.3.gem
logstash-filter-mutate-2.0.2.gem
logstash-filter-ruby-2.0.2.gem
logstash-filter-sleep-2.0.2.gem
logstash-filter-split-2.0.2.gem
logstash-filter-syslog_pri-2.0.2.gem
logstash-filter-throttle-2.0.2.gem
logstash-filter-urldecode-2.0.2.gem
logstash-filter-useragent-2.0.3.gem
logstash-filter-uuid-2.0.3.gem
logstash-filter-xml-2.0.2.gem
logstash-input-beats-2.0.3.gem
logstash-input-couchdb_changes-2.0.2.gem
logstash-input-elasticsearch-2.0.2.gem
logstash-input-eventlog-3.0.1.gem
logstash-input-exec-2.0.4.gem
logstash-input-file-2.0.3.gem
logstash-input-ganglia-2.0.4.gem
logstash-input-gelf-2.0.2.gem
logstash-input-generator-2.0.2.gem
logstash-input-graphite-2.0.4.gem
logstash-input-heartbeat-2.0.2.gem
logstash-input-http-2.0.2.gem
logstash-input-imap-2.0.2.gem
logstash-input-irc-2.0.3.gem
logstash-input-jdbc-2.0.5.gem
logstash-input-kafka-2.0.2.gem
logstash-input-log4j-2.0.4-java.gem
logstash-input-lumberjack-2.0.5.gem
logstash-input-pipe-2.0.2.gem
logstash-input-rabbitmq-3.1.1.gem
logstash-input-redis-2.0.2.gem
logstash-input-s3-2.0.3.gem
logstash-input-snmptrap-2.0.2.gem
logstash-input-sqs-2.0.3.gem
logstash-input-stdin-2.0.2.gem
logstash-input-syslog-2.0.2.gem
logstash-input-tcp-3.0.0.gem
logstash-input-twitter-2.2.0.gem
logstash-input-udp-2.0.3.gem
logstash-input-unix-2.0.4.gem
logstash-input-xmpp-2.0.3.gem
logstash-input-zeromq-2.0.2.gem
logstash-mixin-aws-2.0.2.gem
logstash-mixin-http_client-2.0.3.gem
logstash-mixin-rabbitmq_connection-2.2.0-java.gem
logstash-output-cloudwatch-2.0.2.gem
logstash-output-csv-2.0.2.gem
logstash-output-elasticsearch-2.2.0-java.gem
logstash-output-email-3.0.2.gem
logstash-output-exec-2.0.2.gem
logstash-output-file-2.2.0.gem
logstash-output-ganglia-2.0.2.gem
logstash-output-gelf-2.0.2.gem
logstash-output-graphite-2.0.2.gem
logstash-output-hipchat-3.0.2.gem
logstash-output-http-2.0.5.gem
logstash-output-irc-2.0.2.gem
logstash-output-juggernaut-2.0.2.gem
logstash-output-kafka-2.0.1.gem
logstash-output-lumberjack-2.0.4.gem
logstash-output-nagios-2.0.2.gem
logstash-output-nagios_nsca-2.0.3.gem
logstash-output-null-2.0.2.gem
logstash-output-opentsdb-2.0.2.gem
logstash-output-pagerduty-2.0.2.gem
logstash-output-pipe-2.0.2.gem
logstash-output-rabbitmq-3.0.6-java.gem
logstash-output-redis-2.0.2.gem
logstash-output-sns-3.0.2.gem
logstash-output-sqs-2.0.2.gem
logstash-output-statsd-2.0.4.gem
logstash-output-stdout-2.0.3.gem
logstash-output-tcp-2.0.2.gem
logstash-output-udp-2.0.2.gem
logstash-output-xmpp-2.0.2.gem
logstash-output-zeromq-2.0.2.gem
logstash-patterns-core-2.0.2.gem
lru_redux-1.1.0.gem
mail-2.6.3.gem
manticore-0.4.4-java.gem
march_hare-2.11.0-java.gem
memoizable-0.4.2.gem
method_source-0.8.2.gem
metriks-0.9.9.7.gem
mimemagic-0.3.0.gem
mime-types-2.99.gem
minitar-0.5.4.gem
mongo-2.0.6.gem
msgpack-jruby-1.4.1-java.gem
multi_json-1.11.2.gem
multipart-post-2.0.0.gem
multi_xml-0.5.5.gem
murmurhash3-0.1.6-java.gem
naught-1.1.0.gem
nokogiri-1.6.7-java.gem
octokit-3.8.0.gem
polyglot-0.3.5.gem
pry-0.10.3-java.gem
puma-2.11.3-java.gem
rack-1.6.4.gem
redis-3.2.2.gem
ruby-maven-3.3.8.gem
ruby-maven-libs-3.3.3.gem
rubyzip-1.1.7.gem
rufus-scheduler-3.0.9.gem
sawyer-0.6.0.gem
sequel-4.29.0.gem
simple_oauth-0.3.1.gem
slop-3.6.0.gem
snappy-0.0.12-java.gem
snappy-jars-1.1.0.1.2-java.gem
snmp-1.2.0.gem
spoon-0.0.4.gem
statsd-ruby-1.2.0.gem
stud-0.0.22.gem
thread_safe-0.3.5-java.gem
treetop-1.4.15.gem
twitter-5.15.0.gem
tzinfo-1.2.2.gem
unf-0.1.4-java.gem
user_agent_parser-2.3.0.gem
webhdfs-0.7.4.gem
win32-eventlog-0.6.5.gem
xml-simple-1.1.5.gem
xmpp4r-0.5.gem

I'm not sure if this is just a plugin problem, or a pack problem. Two plugins that we use are experiencing this, and seem to install fine in traditional ways (./plugin install )

bin # ./plugin install --local logstash-output-mongo
Installing logstash-output-mongo
Plugin not found, aborting
ERROR: Installation Aborted, message: Could not find gem 'logstash-output-mongo (>= 0) java' in any of the gem sources listed in your Gemfile or installed on this machine.

Tarball pack

# tar -ztvf plugins_package.tar.gz 
-rw-r--r-- wheel/wheel    8704 2016-01-19 10:23 logstash-output-juggernaut-2.0.2.gem
-rw-r--r-- wheel/wheel   75776 2016-01-19 10:23 elasticsearch-api-1.0.15.gem
-rw-r--r-- wheel/wheel    8704 2016-01-19 10:23 logstash-output-opentsdb-2.0.2.gem
-rw-r--r-- wheel/wheel   10752 2016-01-19 10:23 logstash-output-gelf-2.0.2.gem
-rw-r--r-- wheel/wheel    8704 2016-01-19 10:23 logstash-output-sqs-2.0.2.gem
-rw-r--r-- wheel/wheel   75776 2016-01-19 10:23 twitter-5.15.0.gem
-rw-r--r-- wheel/wheel    9216 2016-01-19 10:23 logstash-input-eventlog-3.0.1.gem
-rw-r--r-- wheel/wheel    8192 2016-01-19 10:23 logstash-filter-uuid-2.0.3.gem
-rw-r--r-- wheel/wheel   18944 2016-01-19 10:23 logstash-codec-collectd-2.0.2.gem
-rw-r--r-- wheel/wheel    8704 2016-01-19 10:23 logstash-output-irc-2.0.2.gem
-rw-r--r-- wheel/wheel    8704 2016-01-19 10:23 logstash-codec-plain-2.0.2.gem
-rw-r--r-- wheel/wheel   15872 2016-01-19 10:23 logstash-input-s3-2.0.3.gem
-rw-r--r-- wheel/wheel   80384 2016-01-19 10:23 cinch-2.3.1.gem
-rw-r--r-- wheel/wheel   15872 2016-01-19 10:23 stud-0.0.22.gem
-rw-r--r-- wheel/wheel  228864 2016-01-19 10:23 rack-1.6.4.gem
-rw-r--r-- wheel/wheel   20992 2016-01-19 10:23 multi_xml-0.5.5.gem
-rw-r--r-- wheel/wheel    9216 2016-01-19 10:23 spoon-0.0.4.gem
-rw-r--r-- wheel/wheel   37888 2016-01-19 10:23 http-cookie-1.0.2.gem
-rw-r--r-- wheel/wheel 1113088 2016-01-19 10:23 snappy-0.0.12-java.gem
-rw-r--r-- wheel/wheel   57344 2016-01-19 10:23 octokit-3.8.0.gem
-rw-r--r-- wheel/wheel 1328640 2016-01-19 10:23 sequel-4.29.0.gem
-rw-r--r-- wheel/wheel    7680 2016-01-19 10:23 logstash-output-null-2.0.2.gem
-rw-r--r-- wheel/wheel   14336 2016-01-19 10:23 logstash-codec-multiline-2.0.4.gem
-rw-r--r-- wheel/wheel   28160 2016-01-19 10:23 logstash-patterns-core-2.0.2.gem
-rw-r--r-- wheel/wheel    8192 2016-01-19 10:23 logstash-output-xmpp-2.0.2.gem
-rw-r--r-- wheel/wheel    8192 2016-01-19 10:23 logstash-output-udp-2.0.2.gem
-rw-r--r-- wheel/wheel  446976 2016-01-19 10:23 edn-1.1.0.gem
-rw-r--r-- wheel/wheel   11776 2016-01-19 10:23 logstash-input-sqs-2.0.3.gem
-rw-r--r-- wheel/wheel 3777024 2016-01-19 10:23 jruby-openssl-0.9.12-java.gem
-rw-r--r-- wheel/wheel   87552 2016-01-19 10:23 logstash-core-2.1.1-java.gem
-rw-r--r-- wheel/wheel    7680 2016-01-19 10:23 gelfd-0.2.0.gem
-rw-r--r-- wheel/wheel   66560 2016-01-19 10:23 geoip-1.6.1.gem
-rw-r--r-- wheel/wheel   25088 2016-01-19 10:23 logstash-input-beats-2.0.3.gem
-rw-r--r-- wheel/wheel   18432 2016-01-19 10:23 filewatch-0.6.7.gem
-rw-r--r-- wheel/wheel  763392 2016-01-19 10:23 aws-sdk-v1-1.66.0.gem
-rw-r--r-- wheel/wheel   23552 2016-01-19 10:23 minitar-0.5.4.gem
-rw-r--r-- wheel/wheel    8192 2016-01-19 10:23 logstash-codec-rubydebug-2.0.4.gem
-rw-r--r-- wheel/wheel   49664 2016-01-19 10:23 bson-3.2.6-java.gem
-rw-r--r-- wheel/wheel   10240 2016-01-19 10:23 logstash-input-imap-2.0.2.gem
-rw-r--r-- wheel/wheel   10240 2016-01-19 10:23 logstash-output-email-3.0.2.gem
-rw-r--r-- wheel/wheel   41984 2016-01-19 10:23 user_agent_parser-2.3.0.gem
-rw-r--r-- wheel/wheel   10240 2016-01-19 10:23 logstash-mixin-aws-2.0.2.gem
-rw-r--r-- wheel/wheel   22528 2016-01-19 10:23 jar-dependencies-0.3.1.gem
-rw-r--r-- wheel/wheel    8192 2016-01-19 10:23 logstash-filter-checksum-2.0.2.gem
-rw-r--r-- wheel/wheel   10240 2016-01-19 10:23 logstash-input-rabbitmq-3.1.1.gem
-rw-r--r-- wheel/wheel   98816 2016-01-19 10:23 rubyzip-1.1.7.gem
-rw-r--r-- wheel/wheel  336384 2016-01-19 10:23 mail-2.6.3.gem
-rw-r--r-- wheel/wheel    7680 2016-01-19 10:23 logstash-output-exec-2.0.2.gem
-rw-r--r-- wheel/wheel  354816 2016-01-19 10:23 mimemagic-0.3.0.gem
-rw-r--r-- wheel/wheel   11776 2016-01-19 10:23 equalizer-0.0.10.gem
-rw-r--r-- wheel/wheel   11264 2016-01-19 10:23 multipart-post-2.0.0.gem
-rw-r--r-- wheel/wheel    8192 2016-01-19 10:23 polyglot-0.3.5.gem
-rw-r--r-- wheel/wheel   12288 2016-01-19 10:23 logstash-filter-kv-2.0.2.gem
-rw-r--r-- wheel/wheel    9216 2016-01-19 10:23 logstash-output-rabbitmq-3.0.6-java.gem
-rw-r--r-- wheel/wheel   75776 2016-01-19 10:23 redis-3.2.2.gem
-rw-r--r-- wheel/wheel   11776 2016-01-19 10:23 logstash-input-syslog-2.0.2.gem
-rw-r--r-- wheel/wheel  453632 2016-01-19 10:23 logstash-input-log4j-2.0.4-java.gem
-rw-r--r-- wheel/wheel   20480 2016-01-19 10:23 hipchat-1.5.2.gem
-rw-r--r-- wheel/wheel  112128 2016-01-19 10:23 domain_name-0.5.25.gem
-rw-r--r-- wheel/wheel   10240 2016-01-19 10:23 logstash-filter-throttle-2.0.2.gem
-rw-r--r-- wheel/wheel   35840 2016-01-19 10:23 faraday-0.9.2.gem
-rw-r--r-- wheel/wheel   11264 2016-01-19 10:23 logstash-input-elasticsearch-2.0.2.gem
-rw-r--r-- wheel/wheel   13312 2016-01-19 10:23 ffi-rzmq-core-1.0.4.gem
-rw-r--r-- wheel/wheel    7680 2016-01-19 10:23 logstash-codec-dots-2.0.2.gem
-rw-r--r-- wheel/wheel   10240 2016-01-19 10:23 logstash-mixin-http_client-2.0.3.gem
-rw-r--r-- wheel/wheel    8192 2016-01-19 10:23 buftok-0.2.0.gem
-rw-r--r-- wheel/wheel   24064 2016-01-19 10:23 atomic-1.1.99-java.gem
-rw-r--r-- wheel/wheel   15872 2016-01-19 10:23 logstash-filter-grok-2.0.2.gem
-rw-r--r-- wheel/wheel   19456 2016-01-19 10:23 naught-1.1.0.gem
-rw-r--r-- wheel/wheel   47104 2016-01-19 10:23 ffi-rzmq-2.0.4.gem
-rw-r--r-- wheel/wheel   11776 2016-01-19 10:23 logstash-output-cloudwatch-2.0.2.gem
-rw-r--r-- wheel/wheel  135680 2016-01-19 10:23 pry-0.10.3-java.gem
-rw-r--r-- wheel/wheel   12288 2016-01-19 10:23 logstash-output-kafka-2.0.1.gem
-rw-r--r-- wheel/wheel 13625344 2016-01-19 10:23 jruby-kafka-1.4.0-java.gem
-rw-r--r-- wheel/wheel    12800 2016-01-19 10:23 logstash-input-twitter-2.2.0.gem
-rw-r--r-- wheel/wheel     9216 2016-01-19 10:23 logstash-filter-json-2.0.2.gem
-rw-r--r-- wheel/wheel     9216 2016-01-19 10:23 logstash-codec-json_lines-2.0.2.gem
-rw-r--r-- wheel/wheel    10240 2016-01-19 10:23 logstash-filter-csv-2.1.0.gem
-rw-r--r-- wheel/wheel    15872 2016-01-19 10:23 sawyer-0.6.0.gem
-rw-r--r-- wheel/wheel   826880 2016-01-19 10:23 msgpack-jruby-1.4.1-java.gem
-rw-r--r-- wheel/wheel   175616 2016-01-19 10:23 xmpp4r-0.5.gem
-rw-r--r-- wheel/wheel     8192 2016-01-19 10:23 logstash-codec-es_bulk-2.0.2.gem
-rw-r--r-- wheel/wheel    38400 2016-01-19 10:23 hitimes-1.2.3-java.gem
-rw-r--r-- wheel/wheel     9216 2016-01-19 10:23 logstash-codec-graphite-2.0.2.gem
-rw-r--r-- wheel/wheel     9728 2016-01-19 10:23 logstash-filter-dns-2.0.2.gem
-rw-r--r-- wheel/wheel     8704 2016-01-19 10:23 logstash-filter-syslog_pri-2.0.2.gem
-rw-r--r-- wheel/wheel     9216 2016-01-19 10:23 logstash-output-lumberjack-2.0.4.gem
-rw-r--r-- wheel/wheel    11776 2016-01-19 10:23 logstash-input-kafka-2.0.2.gem
-rw-r--r-- wheel/wheel   120832 2016-01-19 10:23 puma-2.11.3-java.gem
-rw-r--r-- wheel/wheel   111104 2016-01-19 10:23 concurrent-ruby-0.9.2-java.gem
-rw-r--r-- wheel/wheel    21504 2016-01-19 10:23 logstash-filter-useragent-2.0.3.gem
-rw-r--r-- wheel/wheel    13824 2016-01-19 10:23 logstash-input-jdbc-2.0.5.gem
-rw-r--r-- wheel/wheel     8704 2016-01-19 10:23 logstash-input-graphite-2.0.4.gem
-rw-r--r-- wheel/wheel    13824 2016-01-19 10:23 http-form_data-1.0.1.gem
-rw-r--r-- wheel/wheel     8192 2016-01-19 10:23 logstash-codec-edn_lines-2.0.2.gem
-rw-r--r-- wheel/wheel     8192 2016-01-19 10:23 logstash-mixin-rabbitmq_connection-2.2.0-java.gem
-rw-r--r-- wheel/wheel    11776 2016-01-19 10:23 logstash-output-graphite-2.0.2.gem
-rw-r--r-- wheel/wheel    14336 2016-01-19 10:23 gelf-1.3.2.gem
-rw-r--r-- wheel/wheel     8192 2016-01-19 10:23 logstash-filter-urldecode-2.0.2.gem
-rw-r--r-- wheel/wheel     9216 2016-01-19 10:23 logstash-output-pagerduty-2.0.2.gem
-rw-r--r-- wheel/wheel    14336 2016-01-19 10:23 murmurhash3-0.1.6-java.gem
-rw-r--r-- wheel/wheel    11264 2016-01-19 10:23 logstash-filter-metrics-3.0.0.gem
-rw-r--r-- wheel/wheel    13312 2016-01-19 10:23 ffi-1.9.10-java.gem
-rw-r--r-- wheel/wheel     9728 2016-01-19 10:23 jls-grok-0.11.2.gem
-rw-r--r-- wheel/wheel     8704 2016-01-19 10:23 logstash-output-ganglia-2.0.2.gem
-rw-r--r-- wheel/wheel     8192 2016-01-19 10:23 logstash-codec-msgpack-2.0.2-java.gem
-rw-r--r-- wheel/wheel   256000 2016-01-19 10:23 http_parser.rb-0.6.0-java.gem
-rw-r--r-- wheel/wheel    12288 2016-01-19 10:23 logstash-input-file-2.0.3.gem
-rw-r--r-- wheel/wheel    12800 2016-01-19 10:23 jls-lumberjack-0.0.26.gem
-rw-r--r-- wheel/wheel    10752 2016-01-19 10:23 logstash-output-redis-2.0.2.gem
-rw-r--r-- wheel/wheel     8704 2016-01-19 10:23 logstash-codec-fluent-2.0.2-java.gem
-rw-r--r-- wheel/wheel     9728 2016-01-19 10:23 logstash-output-zeromq-2.0.2.gem
-rw-r--r-- wheel/wheel    56320 2016-01-19 10:23 rufus-scheduler-3.0.9.gem
-rw-r--r-- wheel/wheel     9216 2016-01-19 10:23 logstash-input-unix-2.0.4.gem
-rw-r--r-- wheel/wheel    13312 2016-01-19 10:23 logstash-input-lumberjack-2.0.5.gem
-rw-r--r-- wheel/wheel  1506816 2016-01-19 10:23 jrjackson-0.3.7.gem
-rw-r--r-- wheel/wheel     9728 2016-01-19 10:23 logstash-input-udp-2.0.3.gem
-rw-r--r-- wheel/wheel    11264 2016-01-19 10:23 statsd-ruby-1.2.0.gem
-rw-r--r-- wheel/wheel    20992 2016-01-19 10:23 metriks-0.9.9.7.gem
-rw-r--r-- wheel/wheel    29184 2016-01-19 10:23 childprocess-0.5.8.gem
-rw-r--r-- wheel/wheel    10752 2016-01-19 10:23 logstash-input-zeromq-2.0.2.gem
-rw-r--r-- wheel/wheel    88576 2016-01-19 10:23 backports-3.6.7.gem
-rw-r--r-- wheel/wheel   450560 2016-01-19 10:23 march_hare-2.11.0-java.gem
-rw-r--r-- wheel/wheel    81920 2016-01-19 10:23 httparty-0.13.7.gem
-rw-r--r-- wheel/wheel    32256 2016-01-19 10:23 awesome_print-1.6.1.gem
-rw-r--r-- wheel/wheel    14336 2016-01-19 10:23 logstash-filter-mutate-2.0.2.gem
-rw-r--r-- wheel/wheel    25088 2016-01-19 10:23 multi_json-1.11.2.gem
-rw-r--r-- wheel/wheel    10240 2016-01-19 10:23 logstash-filter-fingerprint-2.0.2.gem
-rw-r--r-- wheel/wheel     8192 2016-01-19 10:23 logstash-input-stdin-2.0.2.gem
-rw-r--r-- wheel/wheel     9216 2016-01-19 10:23 logstash-output-nagios-2.0.2.gem
-rw-r--r-- wheel/wheel    89600 2016-01-19 10:23 json-1.8.3-java.gem
-rw-r--r-- wheel/wheel    33280 2016-01-19 10:23 win32-eventlog-0.6.5.gem
-rw-r--r-- wheel/wheel     8704 2016-01-19 10:23 logstash-filter-ruby-2.0.2.gem
-rw-r--r-- wheel/wheel   147456 2016-01-19 10:23 tzinfo-1.2.2.gem
-rw-r--r-- wheel/wheel     8704 2016-01-19 10:23 logstash-codec-oldlogstashjson-2.0.2.gem
-rw-r--r-- wheel/wheel    71680 2016-01-19 10:23 bindata-2.1.0.gem
-rw-r--r-- wheel/wheel    95232 2016-01-19 10:23 coderay-1.1.0.gem
-rw-r--r-- wheel/wheel     6144 2016-01-19 10:23 filesize-0.0.4.gem
-rw-r--r-- wheel/wheel     9728 2016-01-19 10:23 logstash-input-gelf-2.0.2.gem
-rw-r--r-- wheel/wheel    13312 2016-01-19 10:23 method_source-0.8.2.gem
-rw-r--r-- wheel/wheel     9216 2016-01-19 10:23 logstash-input-exec-2.0.4.gem
-rw-r--r-- wheel/wheel   425472 2016-01-19 10:23 snappy-jars-1.1.0.1.2-java.gem
-rw-r--r-- wheel/wheel    10240 2016-01-19 10:23 logstash-output-sns-3.0.2.gem
-rw-r--r-- wheel/wheel    12288 2016-01-19 10:23 logstash-output-file-2.2.0.gem
-rw-r--r-- wheel/wheel     8704 2016-01-19 10:23 logstash-input-pipe-2.0.2.gem
-rw-r--r-- wheel/wheel    11776 2016-01-19 10:23 file-dependencies-0.1.6.gem
-rw-r--r-- wheel/wheel     9216 2016-01-19 10:23 logstash-output-pipe-2.0.2.gem
-rw-r--r-- wheel/wheel     8704 2016-01-19 10:23 logstash-codec-line-2.0.2.gem
-rw-r--r-- wheel/wheel    12288 2016-01-19 10:23 lru_redux-1.1.0.gem
-rw-r--r-- wheel/wheel    13312 2016-01-19 10:23 logstash-filter-date-2.0.2.gem
-rw-r--r-- wheel/wheel     9216 2016-01-19 10:23 logstash-filter-anonymize-2.0.2.gem
-rw-r--r-- wheel/wheel    84992 2016-01-19 10:23 gems-0.8.3.gem
-rw-r--r-- wheel/wheel   147456 2016-01-19 10:23 mime-types-2.99.gem
-rw-r--r-- wheel/wheel    11776 2016-01-19 10:23 logstash-output-http-2.0.5.gem
-rw-r--r-- wheel/wheel     9728 2016-01-19 10:23 logstash-output-nagios_nsca-2.0.3.gem
-rw-r--r-- wheel/wheel     8192 2016-01-19 10:23 logstash-filter-clone-2.0.4.gem
-rw-r--r-- wheel/wheel    27648 2016-01-19 10:23 logstash-output-elasticsearch-2.2.0-java.gem
-rw-r--r-- wheel/wheel    12800 2016-01-19 10:23 logstash-input-ganglia-2.0.4.gem
-rw-r--r-- wheel/wheel     8704 2016-01-19 10:23 logstash-input-generator-2.0.2.gem
-rw-r--r-- wheel/wheel    60416 2016-01-19 10:23 i18n-0.6.9.gem
-rw-r--r-- wheel/wheel    18944 2016-01-19 10:23 jmespath-1.1.3.gem
-rw-r--r-- wheel/wheel 13905920 2016-01-19 10:23 logstash-filter-geoip-2.0.4.gem
-rw-r--r-- wheel/wheel     8192 2016-01-19 10:23 logstash-output-stdout-2.0.3.gem
-rw-r--r-- wheel/wheel   120832 2016-01-19 10:23 unf-0.1.4-java.gem
-rw-r--r-- wheel/wheel   483328 2016-01-19 10:23 aws-sdk-core-2.1.36.gem
-rw-r--r-- wheel/wheel     9728 2016-01-19 10:23 logstash-codec-json-2.0.4.gem
-rw-r--r-- wheel/wheel    13824 2016-01-19 10:23 logstash-codec-netflow-2.0.2.gem
-rw-r--r-- wheel/wheel    15360 2016-01-19 10:23 arr-pm-0.0.10.gem
-rw-r--r-- wheel/wheel    12800 2016-01-19 10:23 avl_tree-1.2.1.gem
-rw-r--r-- wheel/wheel    23552 2016-01-19 10:23 slop-3.6.0.gem
-rw-r--r-- wheel/wheel  2484224 2016-01-19 10:23 nokogiri-1.6.7-java.gem
-rw-r--r-- wheel/wheel     8704 2016-01-19 10:23 logstash-filter-split-2.0.2.gem
-rw-r--r-- wheel/wheel    24576 2016-01-19 10:23 clamp-0.6.5.gem
-rw-r--r-- wheel/wheel   114176 2016-01-19 10:23 fpm-1.3.3.gem
-rw-r--r-- wheel/wheel     9728 2016-01-19 10:23 logstash-filter-xml-2.0.2.gem
-rw-r--r-- wheel/wheel    18944 2016-01-19 10:23 logstash-input-couchdb_changes-2.0.2.gem
-rw-r--r-- wheel/wheel     9728 2016-01-19 10:23 logstash-output-statsd-2.0.4.gem
-rw-r--r-- wheel/wheel   227840 2016-01-19 10:23 thread_safe-0.3.5-java.gem
-rw-r--r-- wheel/wheel     8704 2016-01-19 10:23 logstash-output-hipchat-3.0.2.gem
-rw-r--r-- wheel/wheel     9728 2016-01-19 10:23 logstash-input-xmpp-2.0.3.gem
-rw-r--r-- wheel/wheel     8704 2016-01-19 10:23 logstash-filter-sleep-2.0.2.gem
-rw-r--r-- wheel/wheel    13312 2016-01-19 10:23 memoizable-0.4.2.gem
-rw-r--r-- wheel/wheel    11776 2016-01-19 10:23 xml-simple-1.1.5.gem
-rw-r--r-- wheel/wheel    78336 2016-01-19 10:23 treetop-1.4.15.gem
-rw-r--r-- wheel/wheel    12288 2016-01-19 10:23 logstash-input-redis-2.0.2.gem
-rw-r--r-- wheel/wheel     8704 2016-01-19 10:23 logstash-input-heartbeat-2.0.2.gem
-rw-r--r-- wheel/wheel     8704 2016-01-19 10:23 simple_oauth-0.3.1.gem
-rw-r--r-- wheel/wheel     7680 2016-01-19 10:23 logstash-filter-drop-2.0.2.gem
-rw-r--r-- wheel/wheel    44032 2016-01-19 10:23 aws-sdk-resources-2.1.36.gem
-rw-r--r-- wheel/wheel     8704 2016-01-19 10:23 ruby-maven-3.3.8.gem
-rw-r--r-- wheel/wheel   205824 2016-01-19 10:23 snmp-1.2.0.gem
-rw-r--r-- wheel/wheel  8104448 2016-01-19 10:23 ruby-maven-libs-3.3.3.gem
-rw-r--r-- wheel/wheel    99840 2016-01-19 10:23 addressable-2.3.8.gem
-rw-r--r-- wheel/wheel    34816 2016-01-19 10:23 elasticsearch-transport-1.0.15.gem
-rw-r--r-- wheel/wheel  1115136 2016-01-19 10:23 manticore-0.4.4-java.gem
-rw-r--r-- wheel/wheel     4608 2016-01-19 10:23 aws-sdk-2.1.36.gem
-rw-r--r-- wheel/wheel     8704 2016-01-19 10:23 logstash-output-tcp-2.0.2.gem
-rw-r--r-- wheel/wheel     8704 2016-01-19 10:23 logstash-output-csv-2.0.2.gem
-rw-r--r-- wheel/wheel    12288 2016-01-19 10:23 logstash-filter-multiline-2.0.3.gem
-rw-r--r-- wheel/wheel    13824 2016-01-19 10:23 logstash-input-tcp-3.0.0.gem
-rw-r--r-- wheel/wheel     7168 2016-01-19 10:23 gmetric-0.1.3.gem
-rw-r--r-- wheel/wheel    13824 2016-01-19 10:23 webhdfs-0.7.4.gem
-rw-r--r-- wheel/wheel    10752 2016-01-19 10:23 logstash-input-http-2.0.2.gem
-rw-r--r-- wheel/wheel     8704 2016-01-19 10:23 logstash-input-snmptrap-2.0.2.gem
-rw-r--r-- wheel/wheel     9728 2016-01-19 10:23 logstash-input-irc-2.0.3.gem
-rw-r--r-- wheel/wheel    20992 2016-01-19 10:23 cabin-0.7.2.gem
-rw-r--r-- wheel/wheel    66560 2016-01-19 10:23 http-0.9.8.gem
-rw-r--r-- wheel/wheel     8192 2016-01-19 10:23 logstash-codec-edn-2.0.2.gem
-rw-r--r-- wheel/wheel     8704 2016-01-19 10:23 elasticsearch-1.0.15.gem
-rw-r--r-- wheel/wheel   174080 2016-01-19 10:23 mongo-2.0.6.gem

This might be related when packing plugins. #3735

purbon commented 8 years ago

Good night, thanks a lot for your report.

Would it be nice to know a bit more about your system to be able to reproduce it, is actually interesting you got the mongo-2.0.6.gem, but not the mongo output.

Can you let me know more about your env?

Thanks a lot.

purbon commented 8 years ago

Good afternoon, just tried your same flow with:

A few important things,

I wonder if you can verify this for your installation, would really need more details to reproduce your issue. Keep in mind one things, the machine that build the package of plugins have to have internet, otherwise this process does not work, is this your source of issue?

robgil commented 8 years ago

So when you run "./plugin pack", do all the dependencies get in to the tarball thats created?

Try this

Next, on the offline box, copy the plugin pack tar.gz file over and install the pack. "./plugin unpack .tar.gz" Then try a ./plugin install --local logstash-output-s3 or webhdfs

robgil commented 8 years ago

I'm also using java 8, centos6 (selinux disabled), and LS 2.1.1-1

Available Packages
Name        : logstash
Arch        : noarch
Epoch       : 1
Version     : 2.1.1
Release     : 1
Size        : 71 M
Repo        : logstash-2.1
Summary     : An extensible logging pipeline
URL         : http://www.elasticsearch.org/overview/logstash/
License     : ASL 2.0
Description : An extensible logging pipeline
robgil commented 8 years ago

There were some errors with the webhdfs dependency specifically to jruby. The webhdfs maintainer released version 8.0.0 which should include fixes for logstash.

https://github.com/kzk/webhdfs/issues/26

nreese commented 8 years ago

I am seeing identical behavior. I can successfully install logstash-input-mongodb plugin, see it is installed with ./plugin list, and use it. But when I do a ./plugin pack, logstash-input-mongodb is not included in the contents of plugins_package.tar.gz.

Has there been any resolution or workarounds for this issue? I need a way of installing the logstash-input-mongodb plugin onto an offline machine.

I am running logstash 2.2.2, java 1.7.0_79, OSX 10.10.3.

ph commented 7 years ago

after reading this issue I believe the problem is the following, I cannot confirm it completely since I am missing some info.

The missing gems mongodb was installed with this command bin/logstash-plugin install logstash-input-monodb***.gem, in the pack workflow theses locally installed plugins where not included in the generated back since the *.gem where not in the cache folder.

The new offline flow from https://github.com/elastic/logstash/pull/6404 which will be bundled in 5.2.0 is taken care of that problem.

I will close this issue, if this ever come up again in 5.2.0 fee free to open a new issue.