search

logstash-plugins / logstash-output-webhdfs

Web hdfs output for logstash
Apache License 2.0
28 stars 25 forks source link

kerberos fail #39

Closed zhangxinlong633 closed 4 years ago

zhangxinlong633 commented 5 years ago

[2019-03-13T16:33:14,906][ERROR][logstash.outputs.webhdfs ] Webhdfs check request failed. (namenode: vm32102:25002, Exception: gss_init_sec_context did not return GSS_S_COMPLETE: Unspecified GSS failure. Minor code may provide more information No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1000) )

zhangxinlong633 commented 5 years ago

output { webhdfs { host => "vm32103" port => 25002 standby_host => "vm32102" standby_port => 25002 user => "hdfs" path => "/huawei/test.log" compression => "none" codec => "json" use_kerberos_auth => true kerberos_keytab => "/root/eoitek/user.keytab" }

whg517 commented 5 years ago

hi @zhangxinlong633

I configured the keytab directory properly. Why is it that when running logstash the file is not found? no such file to load -- gssapi

Can you help me?

[2019-04-30T10:08:09,433][DEBUG][logstash.codecs.line     ] config LogStash::Codecs::Line/@delimiter = "\n"
[2019-04-30T10:08:09,439][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@path = "/user/whg/test-logstash.text"
[2019-04-30T10:08:09,439][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@use_kerberos_auth = true
[2019-04-30T10:08:09,440][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@port = 50070
[2019-04-30T10:08:09,440][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@host = "m1.node.hadoop"
[2019-04-30T10:08:09,440][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@kerberos_keytab = "/home/kevin/keytab/whg.keytab"
[2019-04-30T10:08:09,440][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@id = "976429169b2c71d6e4fa8f4820a6efa81b8d270dc52f99a616cbb433467db226"
[2019-04-30T10:08:09,441][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@user = "whg"
[2019-04-30T10:08:09,441][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@enable_metric = true
[2019-04-30T10:08:09,441][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@codec = <LogStash::Codecs::Line id=>"line_9a0acec0-ea1d-4fdd-8f3e-612bf05d3ea5", enable_metric=>true, charset=>"UTF-8", delimiter=>"\n">
[2019-04-30T10:08:09,442][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@workers = 1
[2019-04-30T10:08:09,442][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@standby_host = false
[2019-04-30T10:08:09,442][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@standby_port = 50070
[2019-04-30T10:08:09,442][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@idle_flush_time = 1
[2019-04-30T10:08:09,442][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@flush_size = 500
[2019-04-30T10:08:09,442][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@open_timeout = 30
[2019-04-30T10:08:09,443][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@read_timeout = 30
[2019-04-30T10:08:09,443][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@use_httpfs = false
[2019-04-30T10:08:09,443][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@single_file_per_thread = false
[2019-04-30T10:08:09,443][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@retry_known_errors = true
[2019-04-30T10:08:09,444][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@retry_interval = 0.5
[2019-04-30T10:08:09,457][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@retry_times = 5
[2019-04-30T10:08:09,458][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@compression = "none"
[2019-04-30T10:08:09,458][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@snappy_bufsize = 32768
[2019-04-30T10:08:09,458][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@snappy_format = "stream"
[2019-04-30T10:08:09,459][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@use_ssl_auth = false
[2019-04-30T10:08:09,493][DEBUG][logstash.javapipeline    ] Starting pipeline {:pipeline_id=>"main"}
warning: thread "[main]-pipeline-manager" terminated with exception (report_on_exception is true):
LoadError: no such file to load -- gssapi
                  require at org/jruby/RubyKernel.java:984
                  require at /usr/local/share/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65
           prepare_client at /usr/local/share/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-webhdfs-3.0.6/lib/logstash/outputs/webhdfs_helper.rb:27
                 register at /usr/local/share/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-webhdfs-3.0.6/lib/logstash/outputs/webhdfs.rb:153
                 register at org/logstash/config/ir/compiler/OutputStrategyExt.java:106
                 register at org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:48
         register_plugins at /usr/local/share/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:191
                     each at org/jruby/RubyArray.java:1792
         register_plugins at /usr/local/share/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:190
  maybe_setup_out_plugins at /usr/local/share/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:445
            start_workers at /usr/local/share/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:203
                      run at /usr/local/share/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:145
                    start at /usr/local/share/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:104
[2019-04-30T10:08:09,758][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2019-04-30T10:08:09,890][DEBUG][logstash.agent           ] Starting puma
[2019-04-30T10:08:09,926][DEBUG][logstash.agent           ] Trying to start WebServer {:port=>9600}
[2019-04-30T10:08:09,935][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2019-04-30T10:08:09,937][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2019-04-30T10:08:10,017][DEBUG][logstash.api.service     ] [api-service] start
[2019-04-30T10:08:10,042][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (LoadError) no such file to load -- gssapi

kevin@ubuntu-kevin:~/test$ ll /home/kevin/keytab/whg.keytab 
-rw-rw-r-- 1 kevin kevin 442 4月  30 09:52 /home/kevin/keytab/whg.keytab
kevin@ubuntu-kevin:~/test$ tail -10 test.conf 
  webhdfs {
    host => "m1.node.hadoop"
    port => 50070
    path => "/user/whg/test-logstash.text"
    user => "whg"
    kerberos_keytab => "/home/kevin/keytab/whg.keytab"
    use_kerberos_auth => true
  }

}

kevin@ubuntu-kevin:~/test$ klist 
Ticket cache: FILE:/tmp/krb5cc_1000
Default principal: whg@EXAMPLE.CN

Valid starting       Expires              Service principal
2019-04-30T09:54:37  2019-05-01T09:54:37  krbtgt/EXAMPLE.CN@EXAMPLE.CN
2019-04-30T09:54:47  2019-05-01T09:54:37  HTTP/m1.node.hadoop@
2019-04-30T09:54:47  2019-05-01T09:54:37  HTTP/m1.node.hadoop@EXAMPLE.CN
whg517 commented 5 years ago

use kerberos webhdfs start error with exception "no such file to load -- gssapi"

According @dengshaochun to the operation of xx, I installed gssapi for logstash and it can be used normally. But only if you get the TGT on the host using kinit. Just like your error message.

But I just wonder, parameters of the incoming kerberos_keytab meaning? When I commented out the parameter, I noticed that logstash worked properly.

However, if you use this method in docker (getting the TGT through kinit), you will be in trouble. All I try again use kdestroy destruction of the current TGT then redirect kerberos_keytab parameters. But it was found to have reported another error. 

[2019-04-30T10:36:39,030][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@path = "/user/whg/test-logstash.text"
[2019-04-30T10:36:39,034][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@use_kerberos_auth = true
[2019-04-30T10:36:39,034][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@port = 50070
[2019-04-30T10:36:39,034][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@host = "m1.node.hadoop"
[2019-04-30T10:36:39,035][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@kerberos_keytab = "/home/kevin/keytab/whg.keytab"
[2019-04-30T10:36:39,035][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@id = "976429169b2c71d6e4fa8f4820a6efa81b8d270dc52f99a616cbb433467db226"
[2019-04-30T10:36:39,035][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@user = "whg"
[2019-04-30T10:36:39,037][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@enable_metric = true
[2019-04-30T10:36:39,037][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@codec = <LogStash::Codecs::Line id=>"line_6a791df9-470b-4d32-a358-9801f129193b", enable_metric=>true, charset=>"UTF-8", delimiter=>"\n">
[2019-04-30T10:36:39,038][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@workers = 1
[2019-04-30T10:36:39,038][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@standby_host = false
[2019-04-30T10:36:39,038][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@standby_port = 50070
[2019-04-30T10:36:39,039][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@idle_flush_time = 1
[2019-04-30T10:36:39,039][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@flush_size = 500
[2019-04-30T10:36:39,039][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@open_timeout = 30
[2019-04-30T10:36:39,039][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@read_timeout = 30
[2019-04-30T10:36:39,040][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@use_httpfs = false
[2019-04-30T10:36:39,040][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@single_file_per_thread = false
[2019-04-30T10:36:39,040][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@retry_known_errors = true
[2019-04-30T10:36:39,041][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@retry_interval = 0.5
[2019-04-30T10:36:39,042][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@retry_times = 5
[2019-04-30T10:36:39,042][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@compression = "none"
[2019-04-30T10:36:39,042][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@snappy_bufsize = 32768
[2019-04-30T10:36:39,043][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@snappy_format = "stream"
[2019-04-30T10:36:39,044][DEBUG][logstash.outputs.webhdfs ] config LogStash::Outputs::WebHdfs/@use_ssl_auth = false
[2019-04-30T10:36:39,079][DEBUG][logstash.javapipeline    ] Starting pipeline {:pipeline_id=>"main"}
[2019-04-30T10:36:39,658][ERROR][logstash.outputs.webhdfs ] Webhdfs check request failed. (namenode: m1.node.hadoop:50070, Exception: undefined method `read_uint32' for #<FFI::MemoryPointer address=0x7f39740901a0 size=4>)
[2019-04-30T10:36:39,665][DEBUG][logstash.outputs.stdout  ] Closing {:plugin=>"LogStash::Outputs::Stdout"}
[2019-04-30T10:36:39,711][ERROR][logstash.javapipeline    ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<WebHDFS::KerberosError: undefined method `read_uint32' for #<FFI::MemoryPointer address=0x7f39740901a0 size=4>>, :backtrace=>["/usr/local/share/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/webhdfs-0.8.0/lib/webhdfs/client_v1.rb:323:in `request'", "/usr/local/share/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/webhdfs-0.8.0/lib/webhdfs/client_v1.rb:275:in `operate_requests'", "/usr/local/share/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/webhdfs-0.8.0/lib/webhdfs/client_v1.rb:138:in `list'", "/usr/local/share/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-webhdfs-3.0.6/lib/logstash/outputs/webhdfs_helper.rb:49:in `test_client'", "/usr/local/share/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-webhdfs-3.0.6/lib/logstash/outputs/webhdfs.rb:155:in `register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:106:in `register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:48:in `register'", "/usr/local/share/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:191:in `block in register_plugins'", "org/jruby/RubyArray.java:1792:in `each'", "/usr/local/share/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:190:in `register_plugins'", "/usr/local/share/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:445:in `maybe_setup_out_plugins'", "/usr/local/share/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:203:in `start_workers'", "/usr/local/share/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:145:in `run'", "/usr/local/share/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:104:in `block in start'"], :thread=>"#<Thread:0x7cac5f13 run>"}
[2019-04-30T10:36:39,738][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2019-04-30T10:36:39,794][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2019-04-30T10:36:39,795][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2019-04-30T10:36:39,834][DEBUG][logstash.instrument.periodicpoller.os] Stopping
[2019-04-30T10:36:39,877][DEBUG][logstash.agent           ] Starting puma
[2019-04-30T10:36:39,883][DEBUG][logstash.instrument.periodicpoller.jvm] Stopping
[2019-04-30T10:36:39,888][DEBUG][logstash.instrument.periodicpoller.persistentqueue] Stopping
[2019-04-30T10:36:39,894][DEBUG][logstash.instrument.periodicpoller.deadletterqueue] Stopping
[2019-04-30T10:36:39,893][DEBUG][logstash.agent           ] Trying to start WebServer {:port=>9600}
[2019-04-30T10:36:39,940][DEBUG][logstash.api.service     ] [api-service] start
[2019-04-30T10:36:40,068][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2019-04-30T10:36:44,971][DEBUG][logstash.agent           ] Shutting down all pipelines {:pipelines_count=>0}
[2019-04-30T10:36:44,981][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>0}
[2019-04-30T10:36:44,987][INFO ][logstash.runner          ] Logstash shut down.

I remember before seen kerberos somewhere about read_uint32 problems, seems to involve the C language, there is no careful study, but now encountered this problem again.