search

OpenTSDB / opentsdb

A scalable, distributed Time Series Database.
http://opentsdb.net
GNU Lesser General Public License v2.1
5k stars 1.25k forks source link

Opentsdb on Mapr error: Deferred.handleContinuation] - callback=shutdown HBase client@1898325501 returned Deferred@2075809815 #1901

Open shivam-dwivedi opened 4 years ago

shivam-dwivedi commented 4 years ago

Hi All, I am trying to implement Opensource OpenTSDB on MaprDB after all the configuration changes.. tsd daemon just shutsdown .. i dont see any major error message. Below are the logs. Appreciate if anyone can help me.

/usr/share/opentsdb/bin/tsdb tsd --port=4242 --staticroot="/usr/share/opentsdb/static/" --cachedir="/tmp/opentsdb" --config=/etc/opentsdb/opentsdb.conf Exception in thread "main" java.lang.RuntimeException: Initialization failed at net.opentsdb.tools.TSDMain.main(TSDMain.java:289) Caused by: java.lang.NullPointerException at net.opentsdb.utils.Config.getBoolean(Config.java:413) at net.opentsdb.tools.TSDMain.main(TSDMain.java:248)

=== Logs

15:57:36.186 INFO [TSDMain.main] - Starting. 15:57:36.189 INFO [TSDMain.main] - net.opentsdb.tools 2.4.0 built at revision 8d13bd5 (MODIFIED) 15:57:36.189 INFO [TSDMain.main] - Built on 2019/10/31 19:09:07 +0000 by root@3d8f599bb923:/root/opensource/mapr-opentsdb-2.4.0/dl/mapr-opentsdb-2.4.0/build 15:57:36.192 INFO [Config.loadConfig] - Successfully loaded configuration file: /etc/opentsdb/opentsdb.conf 15:57:36.199 DEBUG [SelectorUtil.debug] - Using select timeout of 500 15:57:36.199 DEBUG [SelectorUtil.debug] - Epoll-bug workaround enabled = false 15:57:36.258 INFO [Config.loadConfig] - Successfully loaded configuration file: /etc/opentsdb/opentsdb.conf 15:57:36.423 DEBUG [CLDBRpcCommonUtils.init] - init 15:57:36.427 DEBUG [JVMProperties.setPropIfUnset] - Setting JVM property javax.net.ssl.trustStore to /opt/mapr/conf/ssl_truststore 15:57:36.427 DEBUG [JVMProperties.setPropIfUnset] - Setting JVM property http.auth.preference to basic 15:57:36.427 DEBUG [JVMProperties.setPropIfUnset] - Setting JVM property zookeeper.saslprovider to com.mapr.security.maprsasl.MaprSaslProvider 15:57:36.427 DEBUG [JVMProperties.setPropIfUnset] - Setting JVM property java.security.auth.login.config to /opt/mapr/conf/mapr.login.conf 15:57:36.429 DEBUG [Configuration.getProperties] - Loaded org.apache.hadoop.conf.CoreDefaultProperties 15:57:36.432 INFO [MapRThreadPool.] - Creating ThreadPoolExecutor with 128 workerThreads and 5 callbackThreads. 15:57:36.460 DEBUG [HBaseClient.] - Connection Type: NONE 15:57:36.478 DEBUG [Configuration.getProperties] - Loaded org.apache.hadoop.conf.CoreDefaultProperties 15:57:36.532 DEBUG [Shell.isSetsidSupported] - setsid exited with exit code 0 15:57:36.538 DEBUG [Configuration.getProperties] - Loaded org.apache.hadoop.conf.CoreDefaultProperties 15:57:36.572 DEBUG [MutableMetricsFactory.newForField] - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)]) 15:57:36.576 DEBUG [MutableMetricsFactory.newForField] - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)]) 15:57:36.576 DEBUG [MutableMetricsFactory.newForField] - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[GetGroups]) 15:57:36.577 DEBUG [MetricsSystemImpl.register] - UgiMetrics, User and group related metrics 15:57:36.584 DEBUG [Configuration.getProperties] - Loaded org.apache.hadoop.conf.CoreDefaultProperties 15:57:36.585 DEBUG [UserGroupInformation.initialize] - HADOOP_SECURITY_AUTHENTICATION is set to: CUSTOM 15:57:36.591 DEBUG [Configuration.getProperties] - Loaded org.apache.hadoop.conf.CoreDefaultProperties 15:57:36.615 DEBUG [RpcAuthRegistry.addRpcAuthMethod] - Added SIMPLE to registry. 15:57:36.615 DEBUG [RpcAuthRegistry.addRpcAuthMethod] - Added KERBEROS to registry. 15:57:36.615 DEBUG [RpcAuthRegistry.addRpcAuthMethod] - Added TOKEN to registry. 15:57:36.615 DEBUG [RpcAuthRegistry.addRpcAuthMethod] - Added FAKE to registry. 15:57:36.615 DEBUG [RpcAuthRegistry.addRpcAuthMethod] - Added MAPRSASL to registry. 15:57:36.616 DEBUG [UserGroupInformation.initialize] - Security is enabled. 15:57:36.616 DEBUG [UserGroupInformation.initialize] - Login configuration entry is hadoop_default 15:57:36.616 DEBUG [UserGroupInformation.setUGIAuthenticationMethodFromJAASConfiguration] - authenticationMethod from JAAS configuration:CUSTOM 15:57:36.619 DEBUG [Groups.getUserToGroupsMappingService] - Creating new Groups object 15:57:36.621 DEBUG [NativeCodeLoader.] - Trying to load the custom-built native-hadoop library... 15:57:36.621 DEBUG [NativeCodeLoader.] - Loaded the native-hadoop library 15:57:36.621 DEBUG [JniBasedUnixGroupsMapping.] - Using JniBasedUnixGroupsMapping for Group resolution 15:57:36.622 DEBUG [JniBasedUnixGroupsMappingWithFallback.] - Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMapping 15:57:36.624 DEBUG [Groups.] - Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000 15:57:36.630 DEBUG [MaprSecurityLoginModule.login] - Need to addplainticket: false 15:57:36.633 DEBUG [MapRLoginHttpsClient.authenticateIfNeeded] - Entering authenticate if needed. 15:57:36.634 DEBUG [MapRLoginHttpsClient.isKerberosAnOption] - Kerberos not configured for this cluster. 15:57:36.645 DEBUG [MapRLoginHttpsClient.doesSecurityHaveGoodKey] - found existing MapR ticket 15:57:36.646 DEBUG [MapRLoginHttpsClient.authenticateIfNeeded] - Already have good ticket, done 15:57:36.647 DEBUG [HadoopLoginModule.initialize] - Priority principal search list is [class javax.security.auth.kerberos.KerberosPrincipal, class com.mapr.security.MapRPrincipal] 15:57:36.647 DEBUG [HadoopLoginModule.initialize] - Additional principal search list is [class com.sun.security.auth.UnixPrincipal] 15:57:36.647 DEBUG [HadoopLoginModule.login] - hadoop login 15:57:36.647 DEBUG [HadoopLoginModule.commit] - hadoop login commit 15:57:36.648 DEBUG [UserGroupInformation.] - found custom auth principal com.mapr.security.MapRPrincipal in subject. marking authentication method as CUSTOM 15:57:36.649 DEBUG [UserGroupInformation.] - found custom auth principal com.mapr.security.MapRPrincipal in subject. marking authentication method as CUSTOM 15:57:36.649 DEBUG [UserGroupInformation.loginUserFromSubject] - UGI loginUser:mapr (auth:CUSTOM) 15:57:36.686 DEBUG [TSDB.] - TSD Configuration: File [/etc/opentsdb/opentsdb.conf] Key [tsd.core.meta.enable_realtime_uid] Value [false] Key [tsd.storage.hbase.tree_table] Value [/user/mapr/tsdb-tree] Key [tsd.core.meta.cache.enable] Value [false] Key [tsd.core.stats_with_port] Value [false] Key [tsd.http.cachedir] Value [/tmp/opentsdb] Key [tsd.uid.lru.id.size] Value [5000000] Key [tsd.core.tree.enable_processing] Value [false] Key [tsd.query.multi_get.concurrent] Value [20] Key [tsd.storage.use_otsdb_timestamp] Value [false] Key [tsd.query.enable_fuzzy_filter] Value [true] Key [tsd.storage.get_date_tiered_compaction_start] Value [0] Key [tsd.query.skip_unresolved_tagvs] Value [false] Key [tsd.storage.compaction.flush_speed] Value [2] Key [tsd.rollups.enable] Value [false] Key [tsd.timeseriesfilter.enable] Value [false] Key [tsd.no_diediedie] Value [false] Key [tsd.query.limits.overrides.interval] Value [60000] Key [tsd.query.multi_get.limit] Value [131072] Key [tsd.storage.hbase.prefetch_meta] Value [false] Key [tsd.core.meta.enable_tsuid_incrementing] Value [false] Key [tsd.query.filter.expansion_limit] Value [4096] Key [tsd.core.auto_create_metrics] Value [true] Key [tsd.core.plugin_path] Value [/usr/share/opentsdb/plugins] Key [tsd.query.limits.bytes.default] Value [0] Key [tsd.startup.enable] Value [false] Key [tsd.http.request.cors_domains] Value [] Key [tsd.search.plugin] Value [] Key [tsd.query.multi_get.batch_size] Value [1024] Key [tsd.uid.use_mode] Value [false] Key [tsd.storage.fix_duplicates] Value [false] Key [tsd.network.keep_alive] Value [true] Key [tsd.storage.use_max_value] Value [true] Key [tsd.network.port] Value [14242] Key [tsd.core.uid.random_metrics] Value [false] Key [HADOOP_HOME] Value [/opt/mapr/hadoop/hadoop-2.7.0] Key [tsd.query.limits.bytes.allow_override] Value [false] Key [tsd.storage.flush_interval] Value [1000] Key [tsd.storage.enable_compaction] Value [true] Key [tsd.core.bulk.allow_out_of_order_timestamps] Value [false] Key [tsd.storage.hbase.data_table] Value [/tmp/mapr.opentsdb/tsdb] Key [tsd.core.meta.enable_realtime_ts] Value [false] Key [tsd.startup.plugin] Value [] Key [tsd.uid.lru.name.size] Value [5000000] Key [tsd.core.auto_create_tagks] Value [true] Key [tsd.query.limits.data_points.allow_override] Value [false] Key [tsd.rtpublisher.enable] Value [false] Key [tsd.network.worker_threads] Value [] Key [tsd.core.preload_uid_cache.max_entries] Value [300000] Key [tsd.core.authentication.enable] Value [false] Key [tsd.rtpublisher.plugin] Value [] Key [tsd.rollups.block_derived] Value [true] Key [tsd.uidfilter.enable] Value [false] Key [tsd.rollups.agg_tag_key] Value [_aggregate] Key [tsd.network.async_io] Value [true] Key [tsd.core.authentication.plugin] Value [] Key [tsd.core.enable_ui] Value [true] Key [tsd.query.timeout] Value [0] Key [tsd.http.request.max_chunk] Value [4096] Key [tsd.storage.hbase.zk_quorum] Value [localhost:5181] Key [tsd.core.enable_api] Value [true] Key [tsd.http.staticroot] Value [/usr/share/opentsdb/static/] Key [tsd.rpc.telnet.return_errors] Value [true] Key [tsd.query.limits.data_points.default] Value [0] Key [tsd.storage.enable_appends] Value [false] Key [tsd.storage.compaction.min_flush_threshold] Value [100] Key [tsd.network.reuse_address] Value [true] Key [tsd.query.multi_get.get_all_salts] Value [false] Key [tsd.rollups.raw_agg_tag_value] Value [RAW] Key [tsd.http.query.allow_delete] Value [false] Key [tsd.search.enable] Value [false] Key [tsd.http.header_tag] Value [] Key [tsd.storage.hbase.uid_table] Value [/tmp/mapr.opentsdb/tsdb-uid] Key [tsd.storage.hbase.zk_basedir] Value [/hbase] Key [tsd.storage.compaction.max_concurrent_flushes] Value [10000] Key [MAPR_TICKETFILE_LOCATION] Value [/opt/mapr/conf/mapruserticket] Key [tsd.core.auto_create_tagvs] Value [true] Key [tsd.http.show_stack_trace] Value [true] Key [tsd.http.request.enable_chunked] Value [false] Key [tsd.storage.repair_appends] Value [false] Key [tsd.http.request.cors_headers] Value [Authorization, Content-Type, Accept, Origin, User-Agent, DNT, Cache-Control, X-Mx-ReqToken, Keep-Alive, X-Requested-With, If-Modified-Since] Key [tsd.core.socket.timeout] Value [0] Key [tsd.query.allow_simultaneous_duplicates] Value [true] Key [tsd.stats.canonical] Value [false] Key [tsd.uid.lru.enable] Value [false] Key [tsd.network.bind] Value [0.0.0.0] Key [tsd.rollups.tag_raw] Value [false] Key [tsd.storage.hbase.scanner.maxNumRows] Value [128] Key [tsd.network.tcp_no_delay] Value [true] Key [tsd.mode] Value [rw] Key [tsd.query.multi_get.enable] Value [false] Key [tsd.storage.hbase.meta_table] Value [/tmp/mapr.opentsdb/tsdb-meta] Key [tsd.core.connections.limit] Value [0] Key [tsd.core.storage_exception_handler.enable] Value [false] Key [tsd.core.meta.enable_tsuid_tracking] Value [false] Key [tsd.core.preload_uid_cache] Value [false] Key [tsd.storage.compaction.flush_interval] Value [10] 15:57:36.687 DEBUG [PluginLoader.loadJARs] - No JAR files found in path: /usr/share/opentsdb/plugins 15:57:36.689 WARN [PluginLoader.loadPlugins] - Unable to locate any plugins of the type: net.opentsdb.query.filter.TagVFilter 15:57:36.695 DEBUG [Configuration.getProperties] - Loaded org.apache.hadoop.conf.CoreDefaultProperties 15:57:36.702 DEBUG [MapRHTable.init] - Init table /tmp/mapr.opentsdb/tsdb 15:57:36.703 DEBUG [MapRLoginHttpsClient.authenticateIfNeeded] - Entering authenticate if needed. 15:57:36.703 DEBUG [MapRLoginHttpsClient.isKerberosAnOption] - Kerberos not configured for this cluster. 15:57:36.703 DEBUG [MapRLoginHttpsClient.doesSecurityHaveGoodKey] - found existing MapR ticket 15:57:36.703 DEBUG [MapRLoginHttpsClient.authenticateIfNeeded] - Already have good ticket, done 15:57:36.723 DEBUG [MapRFileSystem.lookupClient] - User Info object initialized for user mapr with user ID 5000 15:57:36.729 DEBUG [Inode.debug] - >Inode GetAttr: table: /tmp/mapr.opentsdb/tsdb, size: 3, chunksize: 0, fid: 5445.71.131416 15:57:36.740 DEBUG [MapRHTable.refreshSchema] - Updating schema #families 1 15:57:36.740 DEBUG [MapRHTable.refreshSchema] - Caching family name=t, id=1 15:57:36.741 DEBUG [MapRHTable.init] - opened table /tmp/mapr.opentsdb/tsdb clustername maprclsdevisJson false 15:57:36.743 DEBUG [MapRHTable.init] - Init table /tmp/mapr.opentsdb/tsdb-uid 15:57:36.743 DEBUG [MapRFileSystem.lookupClient] - User Info object initialized for user mapr with user ID 5000 15:57:36.744 DEBUG [Inode.debug] - >Inode GetAttr: table: /tmp/mapr.opentsdb/tsdb-uid, size: 3, chunksize: 0, fid: 5445.74.131422 15:57:36.745 DEBUG [MapRHTable.refreshSchema] - Updating schema #families 2 15:57:36.745 DEBUG [MapRHTable.refreshSchema] - Caching family name=id, id=1 15:57:36.745 DEBUG [MapRHTable.refreshSchema] - Caching family name=name, id=2 15:57:36.745 DEBUG [MapRHTable.init] - opened table /tmp/mapr.opentsdb/tsdb-uid clustername maprclsdevisJson false 15:57:36.746 DEBUG [MapRThreadPool.run] - Calling callback for ThreadId: 286, request: Exists(table="/tmp/mapr.opentsdb/tsdb", key=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], families=null, attempt=0, region=null) 15:57:36.746 DEBUG [MapRThreadPool.run] - Calling callback for ThreadId: 287, request: Exists(table="/tmp/mapr.opentsdb/tsdb-uid", key=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], families=null, attempt=0, region=null) 15:57:36.760 INFO [RpcManager.initializeBuiltinRpcs] - Mode: READWRITE, HTTP UI Enabled: true, HTTP API Enabled: true 15:57:36.770 DEBUG [GnuplotInstaller.] - Inspecting PATH for Gnuplot Exe: [/usr/local/bin] 15:57:36.770 DEBUG [GnuplotInstaller.] - Inspecting PATH for Gnuplot Exe: [/bin] 15:57:36.770 INFO [GnuplotInstaller.] - Found gnuplot at [/bin/gnuplot] 15:57:36.770 INFO [GraphHandler.findGnuplotHelperScript] - Auto Installed Gnuplot Invoker at [/tmp/.tsdb/tsdb-gnuplot/mygnuplot.sh] 15:57:36.780 INFO [RpcHandler.] - TSD is in READWRITE mode 15:57:36.780 INFO [RpcHandler.] - CORS domain list was empty, CORS will not be enabled 15:57:36.781 INFO [RpcHandler.] - Loaded CORS headers (Authorization, Content-Type, Accept, Origin, User-Agent, DNT, Cache-Control, X-Mx-ReqToken, Keep-Alive, X-Requested-With, If-Modified-Since) 15:57:36.783 INFO [ConnectionManager.] - TSD concurrent connection limit set to: 0 15:57:36.785 WARN [PluginLoader.loadPlugins] - Unable to locate any plugins of the type: net.opentsdb.tsd.HttpSerializer 15:57:36.790 INFO [TSDMain.main] - Binding on /0.0.0.0:14242 15:57:36.801 INFO [TSDMain.main] - Ready to serve on /0.0.0.0:14242 15:57:36.812 INFO [TSDB.shutdown] - Flushing compaction queue 15:57:36.815 DEBUG [MapRThreadPool.run] - Calling mTable.asyncFlush(), ThreadId: {} 15:57:36.815 DEBUG [MapRThreadPool.run] - Calling mTable.asyncFlush(), ThreadId: {} 15:57:36.815 DEBUG [MapRThreadPool.run] - Calling callback for ThreadId: 289, request: org.hbase.async.MapRThreadPool$AsyncHBaseRpc@7b5916a8 15:57:36.815 DEBUG [MapRThreadPool.run] - Calling callback for ThreadId: 288, request: org.hbase.async.MapRThreadPool$AsyncHBaseRpc@36cbe7aa 15:57:36.816 DEBUG [Deferred.handleContinuation] - callback=shutdown HBase client@1898325501 returned Deferred@2075809815(state=RUNNING, result=[null, null], callback=net.opentsdb.core.TSDB$1FinalShutdown@742ff096 -> (continuation of Deferred@1967355409 after shutdown HBase client@1898325501), errback=net.opentsdb.core.TSDB$1FinalShutdown@742ff096 -> (continuation of Deferred@1967355409 after shutdown HBase client@1898325501)), so the following Deferred is getting paused: Deferred@1967355409(state=PAUSED, result=Deferred@2075809815, callback=, errback=) 15:57:36.816 DEBUG [HBaseClient.call] - Releasing all remaining resources 15:57:36.817 INFO [TSDB.call] - Completed shutting down the TSDB

manolama commented 3 years ago

Which tsd version/commit are you using? L248 points to a null config object in the loadStartupPlugins() method but I doubt thats it. (and tsd.startup.enable is false in your config).