getting 6org.apache.hadoop.hbase.security.AccessDeniedException\x12\x1AAuthentication is required(\x01" error

[prateekarora-git]

Hi

I am trying to use opentsdb to connect with secure hbase ( kerberos enables) in cloudera CDH5 using below command:

./build/tsdb tsd --port=4242 --staticroot=build/staticroot --cachedir=/tmp/tsdb --zkquorum=myhost:2181

got below error .

2015-04-13 16:16:20,109 INFO [Hashed wheel timer #2-SendThread(master:2181)] ClientCnxn: Session establishment complete on server master/192.168.44.98:2181, sessionid = 0x14ca519006f239b, negotiated timeout = 5000 2015-04-13 16:16:20,112 INFO [Hashed wheel timer #2-EventThread] HBaseClient: Connecting to .META. region @ 192.168.44.99:60020 2015-04-13 16:16:20,116 ERROR [New I/O worker #17] RegionClient: Invalid rpcid: -1 found in ReplayingDecoderBuffer(ridx=99, widx=99)="\x00\x00\x00^\x08\xFF\xFF\xFF\xFF\x0F\x12V\n6org.apache.hadoop.hbase.security.AccessDeniedException\x12\x1AAuthentication is required(\x01" 2015-04-13 16:16:20,116 ERROR [New I/O worker #17] RegionClient: Unexpected exception from downstream on [id: 0x90f2e47a, /192.168.44.103:40530 => /192.168.44.99:60020] org.hbase.async.NonRecoverableException: Invalid rpcid: -1 found in ReplayingDecoderBuffer(ridx=99, widx=99)="\x00\x00\x00^\x08\xFF\xFF\xFF\xFF\x0F\x12V\n6org.apache.hadoop.hbase.security.AccessDeniedException\x12\x1AAuthentication is required(\x01" at org.hbase.async.RegionClient.decode(RegionClient.java:1292) ~[asynchbase-1.6.0.jar:na] at org.hbase.async.RegionClient.decode(RegionClient.java:89) ~[asynchbase-1.6.0.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.9.4.Final.jar:na] at org.hbase.async.RegionClient.handleUpstream(RegionClient.java:1082) ~[asynchbase-1.6.0.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.9.4.Final.jar:na] at org.hbase.async.HBaseClient$RegionClientPipeline.sendUpstream(HBaseClient.java:2677) [asynchbase-1.6.0.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.9.4.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.9.4.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.9.4.Final.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_75] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_75] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] 2015-04-13 16:16:20,117 INFO [New I/O worker #17] HBaseClient: Lost connection with the .META. region

how can i connect opentsdb with secure hbase.

Regards Prateek

[Filix]

I got the same exception with opentsdb v2.0.1

[manolama]

Hi guys, I just pushed the secure AsyncHBase code to the next branch. Compile that, drop it in the proper location, then follow the same steps to setup regular secure HBase client and let me know how it goes please.

[Filix]

Hi @manolama I saw the source code and I think it can't work, because the asynchhbase can't load its config values, hbaseClient.getConfig() only has the default configuration.

TSDB.java

public TSDB(final Config config) {
    this(new HBaseClient(config.getString("tsd.storage.hbase.zk_quorum"),
                         config.getString("tsd.storage.hbase.zk_basedir")),
         config);
  }

HBaseClient.java

public HBaseClient(final String quorum_spec, final String base_path) {
    this(quorum_spec, base_path, defaultChannelFactory());
}

public HBaseClient(final String quorum_spec, final String base_path,
                     final ClientSocketChannelFactory channel_factory) {
    this.channel_factory = channel_factory;
    zkclient = new ZKClient(quorum_spec, base_path);
    config = new Config();  // not specify a config file!!! 
    timer = new HashedWheelTimer(config.getShort("hbase.timer.tick"), 
        MILLISECONDS);
  }

[manolama]

@Filix Yeah we're going to cut a 1.7.0 snapshot and then I can modify OpenTSDB 2.2 to use the new ctor to pass in the proper settings.

[haridsv]

I think I am hitting this same issue. When I looked at the 1.6.0 changes, I found the krb changes already applied, so I was expecting it to work out of the box. Is there a specific patch that I can try on top of 1.6.0 to see if this issue is fixed?

[manolama]

@haridsv With OpenTSDB or the 1.7 client? If the former the next branch (2.2) has been patched to pass the config to asynchbase. For Asynchbase, we never had the security code merged.

[haridsv]

Sorry, I misjudged that 1.6.0 had the security code merged. I have an idea of what patches to apply on top of 1.6.0 and I will try the patched version with opentsdb 2.1.0.

[manolama]

@haridsv Any luck with this? Can we close it out? Thanks!

[haridsv]

Sorry for not updating earlier. Yes, I was able to get it working, so you can close this issue.