use kerberos webhdfs start error with exception "no such file to load -- gssapi"

dengshaochun commented 6 years ago

Please post all product and debugging questions on our forum. Your questions will reach our wider community members there, and if we confirm that there is a bug, then we can open a new issue here.

Logstash Plugins are located in a different organization: https://github.com/logstash-plugins. For bugs on specific Logstash plugins, for example, if Redis Output has a defect, please open it in the respective Redis Output repository.

For all general issues, please provide the following details for fast resolution:

Version:logstash 5.6.3
Operating System:CentOS Linux release 7.1.1503 (Core)
Config File (if you have sensitive info, please remove it):
```
input {
beats {
    port => 5044
}
}
```

output { webhdfs { host => "xxxxxx" port => 50070 path => "/user/hdfs/audit/dt=%{+YYYY-MM-dd}/logstash-%{+HH}.log" user => "hdfs" kerberos_keytab => "/opt/keytab/hdfs.keytab" use_kerberos_auth => "true" standby_host => "xxxxxx" standby_port => 50070 } stdout { codec => rubydebug { metadata => true } } # for debug }

- Sample Data:`log4j logs`
- Steps to Reproduce:
```bash
tar -zxf logstash-5.6.3.tar.gz
cd logstash-5.6.3
bin/logstash-plugin install logstash-output-webhdfs
vi logstash-beat-hdfs.conf
bin/logstash -f logstash-beat-hdfs.conf

Ohter (I try to install gssapi)
```
$ gem list
```

LOCAL GEMS

bigdecimal (1.2.0) ffi (1.9.10) gssapi (1.2.0, 1.1.2) io-console (0.4.2) json (1.7.7) psych (2.0.0) rdoc (4.0.0)

$ vendor/jruby/bin/jruby -S gem install gssapi Successfully installed gssapi-1.2.0 1 gem installed

- Exception:

LoadError: no such file to load -- gssapi require at org/jruby/RubyKernel.java:1040 require at /home/hdfs/logstash/vendor/bundle/jruby/1.9/gems/polyglot-0.3.5/lib/polyglot.rb:65 prepare_client at /home/hdfs/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-webhdfs-3.0.4/lib/logstash/outputs/webhdfs_helper.rb:27 register at /home/hdfs/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-webhdfs-3.0.4/lib/logstash/outputs/webhdfs.rb:146 each at org/jruby/RubyArray.java:1613 register at /home/hdfs/logstash/logstash-core/lib/logstash/output_delegator_strategies/legacy.rb:17 register at /home/hdfs/logstash/logstash-core/lib/logstash/output_delegator.rb:43 register_plugin at /home/hdfs/logstash/logstash-core/lib/logstash/pipeline.rb:290 register_plugins at /home/hdfs/logstash/logstash-core/lib/logstash/pipeline.rb:301 each at org/jruby/RubyArray.java:1613 register_plugins at /home/hdfs/logstash/logstash-core/lib/logstash/pipeline.rb:301 start_workers at /home/hdfs/logstash/logstash-core/lib/logstash/pipeline.rb:310 run at /home/hdfs/logstash/logstash-core/lib/logstash/pipeline.rb:235 start_pipeline at /home/hdfs/logstash/logstash-core/lib/logstash/agent.rb:398

- ruby shell

$ irb irb(main):001:0> require 'gssapi' => true

dengshaochun commented 6 years ago

Solved problem

cp –r /home/hdfs/logstash/vendor/bundle/jruby/1.9/gems/gssapi-1.2.0/lib/* /home/hdfs/logstash/vendor/bundle/jruby/1.9/gems/webhdfs-0.8.0/lib/

but ,I got another problem,when i run logstash

[2017-12-04T16:33:07,969][ERROR][logstash.outputs.webhdfs ] Webhdfs check request failed. (namenode: xxxxxxx:50070, Exception: <html><head><meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/><title>Error 403 GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)</title></head><body><h2>HTTP ERROR 403</h2><p>Problem accessing /webhdfs/v1/. Reason:<pre>    GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)</pre></p><hr /><i><small>Powered by Jetty://</small></i><br/>                                                <br/>                                                <br/>                                                <br/>                                                <br/>                                                <br/>                                                <br/>                                                <br/>                                                <br/>                                                <br/>                                                <br/>                                                <br/>                                                <br/>                                                <br/>                                                <br/>                                                <br/>                                                <br/>                                                <br/>                                                <br/>                                                <br/>                                                </body></html>)

linghu323 commented 6 years ago

I got the same problem, and gssapi can not be installed.

[root@lr14-s01ts logstash-6.2.2]<20180320 12:00:26># vendor/jruby/bin/jruby -S gem install gssapi Successfully installed gssapi-1.2.0 WARNING: Unable to pull data from 'https://rubygems.org/': timed out (https://api.rubygems.org/specs.4.8.gz) 1 gem installed

is it successful?

linghu323 commented 6 years ago

[root@lr14-s02ts logstash-6.2.2]<20180321 16:06:14># bin/logstash -f hdfs.conf Sending Logstash's logs to /opt/logstash/logstash-6.2.2/logs which is now configured via log4j2.properties [2018-03-21T16:06:40,180][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/opt/logstash/logstash-6.2.2/modules/fb_apache/configuration"} [2018-03-21T16:06:40,197][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/opt/logstash/logstash-6.2.2/modules/netflow/configuration"} [2018-03-21T16:06:40,624][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2018-03-21T16:06:41,330][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.2.2"} [2018-03-21T16:06:41,646][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} [2018-03-21T16:06:45,158][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50} [2018-03-21T16:06:46,099][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: LogStash::PipelineAction::Create/pipeline_id:main, action_result: false", :backtrace=>nil} [2018-03-21T16:06:46,202][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: org.jruby.exceptions.RaiseException: (LoadError) no such file to load -- gssapi

irb(main):008:0* require 'gssapi' => true

why?

reilee commented 5 years ago

This issue still not fixed after 2 years?

chitralverma commented 4 years ago

Solved problem

cp –r /home/hdfs/logstash/vendor/bundle/jruby/1.9/gems/gssapi-1.2.0/lib/* /home/hdfs/logstash/vendor/bundle/jruby/1.9/gems/webhdfs-0.8.0/lib/

This worked for me (using the latest logstash and gssapi)!

elastic / logstash

use kerberos webhdfs start error with exception "no such file to load -- gssapi" #8791