search

logstash-plugins / logstash-output-webhdfs

Web hdfs output for logstash
Apache License 2.0
28 stars 26 forks source link

Offline Installs - missing in plugin pack #13

Closed robgil closed 8 years ago

robgil commented 8 years ago

logstash 2.1.1-1 centos 6 logstash-output-webhdfs 2.0.2

When doing a plugin pack, the pack does not contain the logstash-output-webhdfs plugin. It has the webhdfs dep, but not the actual plugin. I'm not completely sure how this ties together. I've also tried installing just the webhdfs dependency that seems to be packaged with the pack generated on the other host. "./plugin list" does not list it as being installed, nor does it list any other deps.

I believe this is an upstream issue with the plugin utility, but opening an issue here for tracking since offline installs are broken.

List of plugins from host that was used to create the pack

The plugins on this host were installed via the typical ./plugin install 

# ./plugin list
logstash-codec-collectd
logstash-codec-dots
logstash-codec-edn
logstash-codec-edn_lines
logstash-codec-es_bulk
logstash-codec-fluent
logstash-codec-graphite
logstash-codec-json
logstash-codec-json_lines
logstash-codec-line
logstash-codec-msgpack
logstash-codec-multiline
logstash-codec-netflow
logstash-codec-oldlogstashjson
logstash-codec-plain
logstash-codec-rubydebug
logstash-filter-anonymize
logstash-filter-checksum
logstash-filter-clone
logstash-filter-csv
logstash-filter-date
logstash-filter-dns
logstash-filter-drop
logstash-filter-fingerprint
logstash-filter-geoip
logstash-filter-grok
logstash-filter-json
logstash-filter-kv
logstash-filter-metrics
logstash-filter-multiline
logstash-filter-mutate
logstash-filter-ruby
logstash-filter-sleep
logstash-filter-split
logstash-filter-syslog_pri
logstash-filter-throttle
logstash-filter-urldecode
logstash-filter-useragent
logstash-filter-uuid
logstash-filter-xml
logstash-input-beats
logstash-input-couchdb_changes
logstash-input-elasticsearch
logstash-input-eventlog
logstash-input-exec
logstash-input-file
logstash-input-ganglia
logstash-input-gelf
logstash-input-generator
logstash-input-graphite
logstash-input-heartbeat
logstash-input-http
logstash-input-imap
logstash-input-irc
logstash-input-jdbc
logstash-input-kafka
logstash-input-log4j
logstash-input-lumberjack
logstash-input-pipe
logstash-input-rabbitmq
logstash-input-redis
logstash-input-s3
logstash-input-snmptrap
logstash-input-sqs
logstash-input-stdin
logstash-input-syslog
logstash-input-tcp
logstash-input-twitter
logstash-input-udp
logstash-input-unix
logstash-input-xmpp
logstash-input-zeromq
logstash-output-cloudwatch
logstash-output-csv
logstash-output-elasticsearch
logstash-output-email
logstash-output-exec
logstash-output-file
logstash-output-ganglia
logstash-output-gelf
logstash-output-graphite
logstash-output-hipchat
logstash-output-http
logstash-output-irc
logstash-output-juggernaut
logstash-output-kafka
logstash-output-lumberjack
logstash-output-mongodb
logstash-output-nagios
logstash-output-nagios_nsca
logstash-output-null
logstash-output-opentsdb
logstash-output-pagerduty
logstash-output-pipe
logstash-output-rabbitmq
logstash-output-redis
logstash-output-s3
logstash-output-sns
logstash-output-sqs
logstash-output-statsd
logstash-output-stdout
logstash-output-tcp
logstash-output-udp
logstash-output-webhdfs
logstash-output-xmpp
logstash-output-zeromq
logstash-patterns-core
# ./plugin pack
Packaging plugins for offline usage
Generated at /opt/logstash/plugins_package.tar.gz

./plugin still tries to hit the interwebs even with --local and --no-verify

bin # ./plugin install --no-verify --local webhdfs
Installing webhdfs
Error Bundler::HTTPError, retrying 1/10
Could not fetch specs from https://rubygems.org/
Error Bundler::HTTPError, retrying 2/10
Could not fetch specs from https://rubygems.org/
Error Bundler::HTTPError, retrying 3/10
Could not fetch specs from https://rubygems.org/
Error Bundler::HTTPError, retrying 4/10
Could not fetch specs from https://rubygems.org/
Error Bundler::HTTPError, retrying 5/10
Could not fetch specs from https://rubygems.org/
Error Bundler::HTTPError, retrying 6/10
Could not fetch specs from https://rubygems.org/
Error Bundler::HTTPError, retrying 7/10
Could not fetch specs from https://rubygems.org/
Error Bundler::HTTPError, retrying 8/10
Could not fetch specs from https://rubygems.org/
Error Bundler::HTTPError, retrying 9/10
Could not fetch specs from https://rubygems.org/
Error Bundler::HTTPError, retrying 10/10
Could not fetch specs from https://rubygems.org/
Too many retries, aborting, caused by Bundler::HTTPError
ERROR: Installation Aborted, message: Could not fetch specs from https://rubygems.org/

List of gems from pack file in /opt/logstash/vendor/cache/

cache # ls -1
addressable-2.3.8.gem
arr-pm-0.0.10.gem
atomic-1.1.99-java.gem
avl_tree-1.2.1.gem
awesome_print-1.6.1.gem
aws-sdk-2.1.36.gem
aws-sdk-core-2.1.36.gem
aws-sdk-resources-2.1.36.gem
aws-sdk-v1-1.66.0.gem
backports-3.6.7.gem
bindata-2.1.0.gem
bson-3.2.6-java.gem
buftok-0.2.0.gem
cabin-0.7.2.gem
childprocess-0.5.8.gem
cinch-2.3.1.gem
clamp-0.6.5.gem
coderay-1.1.0.gem
concurrent-ruby-0.9.2-java.gem
domain_name-0.5.25.gem
edn-1.1.0.gem
elasticsearch-1.0.15.gem
elasticsearch-api-1.0.15.gem
elasticsearch-transport-1.0.15.gem
equalizer-0.0.10.gem
faraday-0.9.2.gem
ffi-1.9.10-java.gem
ffi-rzmq-2.0.4.gem
ffi-rzmq-core-1.0.4.gem
file-dependencies-0.1.6.gem
filesize-0.0.4.gem
filewatch-0.6.7.gem
fpm-1.3.3.gem
gelf-1.3.2.gem
gelfd-0.2.0.gem
gems-0.8.3.gem
geoip-1.6.1.gem
gmetric-0.1.3.gem
hipchat-1.5.2.gem
hitimes-1.2.3-java.gem
http-0.9.8.gem
httparty-0.13.7.gem
http-cookie-1.0.2.gem
http-form_data-1.0.1.gem
http_parser.rb-0.6.0-java.gem
i18n-0.6.9.gem
jar-dependencies-0.3.1.gem
jls-grok-0.11.2.gem
jls-lumberjack-0.0.26.gem
jmespath-1.1.3.gem
jrjackson-0.3.7.gem
jruby-kafka-1.4.0-java.gem
jruby-openssl-0.9.12-java.gem
json-1.8.3-java.gem
logstash-codec-collectd-2.0.2.gem
logstash-codec-dots-2.0.2.gem
logstash-codec-edn-2.0.2.gem
logstash-codec-edn_lines-2.0.2.gem
logstash-codec-es_bulk-2.0.2.gem
logstash-codec-fluent-2.0.2-java.gem
logstash-codec-graphite-2.0.2.gem
logstash-codec-json-2.0.4.gem
logstash-codec-json_lines-2.0.2.gem
logstash-codec-line-2.0.2.gem
logstash-codec-msgpack-2.0.2-java.gem
logstash-codec-multiline-2.0.4.gem
logstash-codec-netflow-2.0.2.gem
logstash-codec-oldlogstashjson-2.0.2.gem
logstash-codec-plain-2.0.2.gem
logstash-codec-rubydebug-2.0.4.gem
logstash-core-2.1.1-java.gem
logstash-filter-anonymize-2.0.2.gem
logstash-filter-checksum-2.0.2.gem
logstash-filter-clone-2.0.4.gem
logstash-filter-csv-2.1.0.gem
logstash-filter-date-2.0.2.gem
logstash-filter-dns-2.0.2.gem
logstash-filter-drop-2.0.2.gem
logstash-filter-fingerprint-2.0.2.gem
logstash-filter-geoip-2.0.4.gem
logstash-filter-grok-2.0.2.gem
logstash-filter-json-2.0.2.gem
logstash-filter-kv-2.0.2.gem
logstash-filter-metrics-3.0.0.gem
logstash-filter-multiline-2.0.3.gem
logstash-filter-mutate-2.0.2.gem
logstash-filter-ruby-2.0.2.gem
logstash-filter-sleep-2.0.2.gem
logstash-filter-split-2.0.2.gem
logstash-filter-syslog_pri-2.0.2.gem
logstash-filter-throttle-2.0.2.gem
logstash-filter-urldecode-2.0.2.gem
logstash-filter-useragent-2.0.3.gem
logstash-filter-uuid-2.0.3.gem
logstash-filter-xml-2.0.2.gem
logstash-input-beats-2.0.3.gem
logstash-input-couchdb_changes-2.0.2.gem
logstash-input-elasticsearch-2.0.2.gem
logstash-input-eventlog-3.0.1.gem
logstash-input-exec-2.0.4.gem
logstash-input-file-2.0.3.gem
logstash-input-ganglia-2.0.4.gem
logstash-input-gelf-2.0.2.gem
logstash-input-generator-2.0.2.gem
logstash-input-graphite-2.0.4.gem
logstash-input-heartbeat-2.0.2.gem
logstash-input-http-2.0.2.gem
logstash-input-imap-2.0.2.gem
logstash-input-irc-2.0.3.gem
logstash-input-jdbc-2.0.5.gem
logstash-input-kafka-2.0.2.gem
logstash-input-log4j-2.0.4-java.gem
logstash-input-lumberjack-2.0.5.gem
logstash-input-pipe-2.0.2.gem
logstash-input-rabbitmq-3.1.1.gem
logstash-input-redis-2.0.2.gem
logstash-input-s3-2.0.3.gem
logstash-input-snmptrap-2.0.2.gem
logstash-input-sqs-2.0.3.gem
logstash-input-stdin-2.0.2.gem
logstash-input-syslog-2.0.2.gem
logstash-input-tcp-3.0.0.gem
logstash-input-twitter-2.2.0.gem
logstash-input-udp-2.0.3.gem
logstash-input-unix-2.0.4.gem
logstash-input-xmpp-2.0.3.gem
logstash-input-zeromq-2.0.2.gem
logstash-mixin-aws-2.0.2.gem
logstash-mixin-http_client-2.0.3.gem
logstash-mixin-rabbitmq_connection-2.2.0-java.gem
logstash-output-cloudwatch-2.0.2.gem
logstash-output-csv-2.0.2.gem
logstash-output-elasticsearch-2.2.0-java.gem
logstash-output-email-3.0.2.gem
logstash-output-exec-2.0.2.gem
logstash-output-file-2.2.0.gem
logstash-output-ganglia-2.0.2.gem
logstash-output-gelf-2.0.2.gem
logstash-output-graphite-2.0.2.gem
logstash-output-hipchat-3.0.2.gem
logstash-output-http-2.0.5.gem
logstash-output-irc-2.0.2.gem
logstash-output-juggernaut-2.0.2.gem
logstash-output-kafka-2.0.1.gem
logstash-output-lumberjack-2.0.4.gem
logstash-output-nagios-2.0.2.gem
logstash-output-nagios_nsca-2.0.3.gem
logstash-output-null-2.0.2.gem
logstash-output-opentsdb-2.0.2.gem
logstash-output-pagerduty-2.0.2.gem
logstash-output-pipe-2.0.2.gem
logstash-output-rabbitmq-3.0.6-java.gem
logstash-output-redis-2.0.2.gem
logstash-output-sns-3.0.2.gem
logstash-output-sqs-2.0.2.gem
logstash-output-statsd-2.0.4.gem
logstash-output-stdout-2.0.3.gem
logstash-output-tcp-2.0.2.gem
logstash-output-udp-2.0.2.gem
logstash-output-xmpp-2.0.2.gem
logstash-output-zeromq-2.0.2.gem
logstash-patterns-core-2.0.2.gem
lru_redux-1.1.0.gem
mail-2.6.3.gem
manticore-0.4.4-java.gem
march_hare-2.11.0-java.gem
memoizable-0.4.2.gem
method_source-0.8.2.gem
metriks-0.9.9.7.gem
mimemagic-0.3.0.gem
mime-types-2.99.gem
minitar-0.5.4.gem
mongo-2.0.6.gem
msgpack-jruby-1.4.1-java.gem
multi_json-1.11.2.gem
multipart-post-2.0.0.gem
multi_xml-0.5.5.gem
murmurhash3-0.1.6-java.gem
naught-1.1.0.gem
nokogiri-1.6.7-java.gem
octokit-3.8.0.gem
polyglot-0.3.5.gem
pry-0.10.3-java.gem
puma-2.11.3-java.gem
rack-1.6.4.gem
redis-3.2.2.gem
ruby-maven-3.3.8.gem
ruby-maven-libs-3.3.3.gem
rubyzip-1.1.7.gem
rufus-scheduler-3.0.9.gem
sawyer-0.6.0.gem
sequel-4.29.0.gem
simple_oauth-0.3.1.gem
slop-3.6.0.gem
snappy-0.0.12-java.gem
snappy-jars-1.1.0.1.2-java.gem
snmp-1.2.0.gem
spoon-0.0.4.gem
statsd-ruby-1.2.0.gem
stud-0.0.22.gem
thread_safe-0.3.5-java.gem
treetop-1.4.15.gem
twitter-5.15.0.gem
tzinfo-1.2.2.gem
unf-0.1.4-java.gem
user_agent_parser-2.3.0.gem
webhdfs-0.7.4.gem
win32-eventlog-0.6.5.gem
xml-simple-1.1.5.gem
xmpp4r-0.5.gem
dstore-dbap commented 8 years ago

Hi robgil,

thanks for your feedback. I had a quick look at it and, as you already wrote, I think it might be a problem with the bundling. Will follow the other issues you opened concerning your problem. If it turns out that this plugin needs bugfixing, I will update this ticket.

Thanks again and take care, Björn

dstore-dbap commented 8 years ago

Hi robgil,

would like to close this issue as it really seems to be a problem with packaging and not with this plugin itself. Please let me know if this works for you.

Thanks and take care, Björn

dstore-dbap commented 8 years ago

Since I didn't hear anything about this issue anymore, I'm going to close this now. If the problem still persists, please open a new issue.

Thanks and take care, Björn