search

kzk / webhdfs

Ruby client for Hadoop WebHDFS
Other
81 stars 46 forks source link

when renew token, handle other corner case #41

Closed JiHyunSong closed 3 years ago

JiHyunSong commented 3 years ago

I got some exception while using delegation token. maybe It occurs from the difference of hadoop version

(it is lower version than before https://github.com/kzk/webhdfs/pull/39#discussion_r679552033) 

{"RemoteException":{"exception":"SecurityException","javaClassName":"java.lang.SecurityException","message":"Failed to obtain user group information: org.apache.hadoop.security.token.SecretManager$InvalidToken: token (token for {USER}: HDFS_DELEGATION_TOKEN owner={USER}, renewer={USER}, realUser=, issueDate=1628742694813, maxDate=1629347494813, sequenceNumber=-2097289115, masterKeyId=1386) is expired, current time: 2021-08-19 13:31:41,396+0900 expected renewal time: 2021-08-19 13:31:34,813+0900"}}

=> It was deleted by namenode because of time expired

in fluentd log

2021-08-19 13:31:41 +0900 [warn]: #1 [webhdfs_hadoop] failed to flush the buffer. retry_time=0 next_retry_seconds=2021-08-19 13:31:42 +0900 chunk="5c9e1cc8f4da71f71f98206c880079d6" error_class=WebHDFS::IOError error="{\"RemoteException\":{\"exception\":\"SecurityException\",\"javaClassName\":\"java.lang.SecurityException\",\"message\":\"Failed to obtain user group information: org.apache.hadoop.security.token.SecretManager$InvalidToken: token (token for {USER}: HDFS_DELEGATION_TOKEN owner={USER}, renewer={USER}, realUser=, issueDate=1628742694813, maxDate=1629347494813, sequenceNumber=-2097289115, masterKeyId=1386) is expired, current time: 2021-08-19 13:31:41,396+0900 expected renewal time: 2021-08-19 13:31:34,813+0900\"}}"
  2021-08-19 13:31:41 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/webhdfs-0.10.1/lib/webhdfs/client_v1.rb:464:in `request'
  2021-08-19 13:31:41 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/webhdfs-0.10.1/lib/webhdfs/client_v1.rb:309:in `operate_requests'
  2021-08-19 13:31:41 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/webhdfs-0.10.1/lib/webhdfs/client_v1.rb:99:in `create'
  2021-08-19 13:31:41 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-webhdfs-1.5.0/lib/fluent/plugin/out_webhdfs.rb:286:in `send_data'
  2021-08-19 13:31:41 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-webhdfs-1.5.0/lib/fluent/plugin/out_webhdfs.rb:411:in `block in write'
  2021-08-19 13:31:41 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-webhdfs-1.5.0/lib/fluent/plugin/out_webhdfs.rb:357:in `compress_context'
  2021-08-19 13:31:41 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-webhdfs-1.5.0/lib/fluent/plugin/out_webhdfs.rb:410:in `write'
  2021-08-19 13:31:41 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/plugin/output.rb:1133:in `try_flush'
  2021-08-19 13:31:41 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/plugin/output.rb:1439:in `flush_thread_run'
  2021-08-19 13:31:41 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/plugin/output.rb:461:in `block (2 levels) in start'
  2021-08-19 13:31:41 +0900 [warn]: #1 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

Signed-off-by: JiHyunSong jihyun.song0829@gmail.com

JiHyunSong commented 3 years ago

@tagomoris check it plz 🙏

above exception occurs from CDH 5.13.1 (hadoop-2.6.0+cdh5.13.1+2666 ref ) when I test before, I'm using vanilla Hadoop 2.10.0

tagomoris commented 3 years ago

Shipped v0.10.2.