search

OpenTSDB / opentsdb

A scalable, distributed Time Series Database.
http://opentsdb.net
GNU Lesser General Public License v2.1
4.99k stars 1.25k forks source link

opentsdb restart fails #636

Open utkarshcmu opened 8 years ago

utkarshcmu commented 8 years ago

Opentsdb restart is failing but it does start the process again. :(

Command output:

sudo service opentsdb restart
Stopping opentsdb:                                         [FAILED]
Starting opentsdb:                                         [  OK  ]

Some logs:

00:29:33.234 INFO  [HttpQuery.logInfo] - [id: 0x23037366, /10.241.63.33:46617 => /10.241.62.217:4242] HTTP /api/put?details done in 1ms
00:29:33.244 INFO  [HttpQuery.logInfo] - [id: 0x23037366, /10.241.63.33:46617 => /10.241.62.217:4242] HTTP /api/put?details done in 1ms
00:29:33.251 INFO  [HttpQuery.logInfo] - [id: 0x23037366, /10.241.63.33:46617 => /10.241.62.217:4242] HTTP /api/put?details done in 1ms
00:29:34.316 INFO  [TSDMain.main] - Starting.
00:29:34.321 INFO  [TSDMain.main] - net.opentsdb.tools 2.2.0RC3 built at revision  (MODIFIED)
00:29:34.321 INFO  [TSDMain.main] - Built on 2015/11/11 21:03:33 +0000 by root@centos.localhost:/root/rpmbuild/BUILD/opentsdb-2.2.0RC3
00:29:34.325 INFO  [Config.loadConfig] - Successfully loaded configuration file: /etc/opentsdb/opentsdb.conf
00:29:34.362 INFO  [Config.loadConfig] - Successfully loaded configuration file: /etc/opentsdb/opentsdb.conf
00:29:34.442 WARN  [PluginLoader.loadPlugins] - Unable to locate any plugins of the type: net.opentsdb.query.filter.TagVFilter
00:29:34.467 INFO  [ZooKeeper.logEnv] - Client environment:zookeeper.version=3.3.6-1366786, built on 07/29/2012 06:22 GMT
manolama commented 8 years ago

Hmm, may be a bad init script. Centos and installed via the RPM package?

utkarshcmu commented 8 years ago

I found that when a tsdb server is overloaded with a lot of requests, the init script shows "failed" msg as above. Otherwise restart happens properly.

manolama commented 8 years ago

Ah if the TSD is busy then yeah, it'll take a while to shutdown. Looks like that killproc utility won't wait very long. Try https://github.com/manolama/opentsdb/tree/i636 and let me know if it works please.

Pengxijin commented 8 years ago

I also have this problem. I intstall opentsdb-2.2.0.noarch.rpm on Centos6.7 64bit.

Command output:

sudo service opentsdb restart
Stopping opentsdb:                                         [FAILED]
Starting opentsdb:                                         [  OK  ]

logs:


2016-05-25 13:36:00,411 INFO  [main] TSDMain: Starting.
2016-05-25 13:36:00,416 INFO  [main] TSDMain: net.opentsdb.tools 2.2.0 built at revision  (MODIFIED)
2016-05-25 13:36:00,417 INFO  [main] TSDMain: Built on 2016/02/14 13:22:24 +0000 by root@centos.localhost:/root/rpmbuild/BUILD/opentsdb-2.2.0
2016-05-25 13:36:00,421 INFO  [main] Config: Successfully loaded configuration file: /etc/opentsdb/opentsdb.conf
2016-05-25 13:36:00,462 INFO  [main] Config: Successfully loaded configuration file: /etc/opentsdb/opentsdb.conf
2016-05-25 13:36:00,553 WARN  [main] PluginLoader: Unable to locate any plugins of the type: net.opentsdb.query.filter.TagVFilter
2016-05-25 13:36:00,553 WARN  [main] PluginLoader: Unable to locate any plugins of the type: net.opentsdb.tsd.RTPublisher
2016-05-25 13:36:00,557 INFO  [main] TSDB: Flushing compaction queue
2016-05-25 13:36:00,573 INFO  [main] TSDB: Completed shutting down the TSDB
Exception in thread "main" java.lang.RuntimeException: Initialization failed
        at net.opentsdb.tools.TSDMain.main(TSDMain.java:196)
Caused by: java.lang.IllegalArgumentException: Unable to locate real time publisher plugin: com.tsdb.kafka.plugin.KafkaProducerRPCPlugin
        at net.opentsdb.core.TSDB.initializePlugins(TSDB.java:294)
        at net.opentsdb.tools.TSDMain.main(TSDMain.java:151)
johann8384 commented 8 years ago

@Pengxijin You seem to have a different problem, have you improperly configured the realtime plugin? It looks like you have tried to enable Easemob's Kafka RPC plugin without providing the plugin.

Try setting "tsd.rtpublisher.plugin" to "" and "tsd.rtpublisher.enable" to "false".

If you want to use that plugin, make sure that is available in the path specified by "tsd.core.plugin_path".