chenghao2262
commented
6 years ago I tracked the stack and found all IO woekrs were waiting. There is no other IO worker can handle http connection. So ConnectionManager always show OPEN.
code in getId(String name): getIdAsync(name).joinUninterruptibly(); Opentsdb will wait without timeout interuption.
HBase crashed once, but we soon rebooted.
But, TSDB didn't recover.
manolama
What happend to my opentsdb? Opentsdb interface cannot work, but the log has no error, and the last log always shows ConnectionManager.handleUpstream Open. There is no other message in log file.
Can any one tell me How to fix this error?
18:33:31.797 INFO [ConnectionManager.handleUpstream] - [id: 0xbf3a82ec, /172.16.97.0:43748 => /172.16.38.5:4242] OPEN 18:33:31.946 INFO [ConnectionManager.handleUpstream] - [id: 0xf2721f5c, /172.16.97.0:43756 => /172.16.38.5:4242] OPEN 18:33:31.970 INFO [ConnectionManager.handleUpstream] - [id: 0xea751419, /172.16.97.0:46522 => /172.16.38.5:4242] OPEN 18:33:32.059 INFO [ConnectionManager.handleUpstream] - [id: 0xa68c593a, /172.16.97.0:43760 => /172.16.38.5:4242] OPEN 18:33:32.121 INFO [ConnectionManager.handleUpstream] - [id: 0xf10f6137, /172.16.97.0:43764 => /172.16.38.5:4242] OPEN 18:33:32.135 INFO [ConnectionManager.handleUpstream] - [id: 0xd058fa90, /172.16.97.0:43766 => /172.16.38.5:4242] OPEN 18:33:32.141 INFO [ConnectionManager.handleUpstream] - [id: 0x37ff6595, /172.16.97.0:43768 => /172.16.38.5:4242] OPEN 18:33:32.352 INFO [ConnectionManager.handleUpstream] - [id: 0x8429c57b, /172.16.97.0:10844 => /172.16.38.5:4242] OPEN 18:33:32.681 INFO [ConnectionManager.handleUpstream] - [id: 0xd5bc418b, /172.16.97.0:43778 => /172.16.38.5:4242] OPEN 18:33:32.891 INFO [ConnectionManager.handleUpstream] - [id: 0x564f90d6, /172.16.97.0:43786 => /172.16.38.5:4242] OPEN 18:33:32.984 INFO [ConnectionManager.handleUpstream] - [id: 0x845103a7, /172.16.97.0:46586 => /172.16.38.5:4242] OPEN 18:33:32.984 INFO [ConnectionManager.handleUpstream] - [id: 0xae35d4ff, /172.16.97.0:43794 => /172.16.38.5:4242] OPEN 18:33:33.293 INFO [ConnectionManager.handleUpstream] - [id: 0x66c93320, /172.16.97.0:43798 => /172.16.38.5:4242] OPEN 18:33:33.789 INFO [ConnectionManager.handleUpstream] - [id: 0x48dab3ce, /172.16.97.0:43804 => /172.16.38.5:4242] OPEN 18:33:33.892 INFO [ConnectionManager.handleUpstream] - [id: 0x8959d4a0, /172.16.97.0:43810 => /172.16.38.5:4242] OPEN 18:33:33.960 INFO [ConnectionManager.handleUpstream] - [id: 0x82e641e9, /172.16.97.0:43818 => /172.16.38.5:4242] OPEN 18:33:33.970 INFO [ConnectionManager.handleUpstream] - [id: 0x6e006add, /172.16.97.0:43820 => /172.16.38.5:4242] OPEN 18:33:34.032 INFO [ConnectionManager.handleUpstream] - [id: 0x5c809aa2, /172.16.97.0:43828 => /172.16.38.5:4242] OPEN 18:33:34.044 INFO [ConnectionManager.handleUpstream] - [id: 0x10bc169e, /172.16.97.0:43830 => /172.16.38.5:4242] OPEN 18:33:34.044 INFO [ConnectionManager.handleUpstream] - [id: 0xdc32dcf9, /172.16.97.0:43832 => /172.16.38.5:4242] OPEN 18:33:34.091 INFO [ConnectionManager.handleUpstream] - [id: 0x3d98061e, /172.16.97.0:43838 => /172.16.38.5:4242] OPEN 18:33:34.337 INFO [ConnectionManager.handleUpstream] - [id: 0xa93ce77d, /172.16.97.0:43842 => /172.16.38.5:4242] OPEN 18:33:34.373 INFO [ConnectionManager.handleUpstream] - [id: 0xf808e9ec, /172.16.97.0:46676 => /172.16.38.5:4242] OPEN 18:33:36.283 INFO [ConnectionManager.handleUpstream] - [id: 0x8e225221, /172.16.97.0:36604 => /172.16.38.5:4242] OPEN 18:33:36.284 INFO [ConnectionManager.handleUpstream] - [id: 0xffeeb5b7, /172.16.97.0:36606 => /172.16.38.5:4242] OPEN 18:33:36.298 INFO [ConnectionManager.handleUpstream] - [id: 0x3f2b103e, /172.16.97.0:10845 => /172.16.38.5:4242] OPEN 18:33:37.109 INFO [ConnectionManager.handleUpstream] - [id: 0x35509da9, /172.16.97.0:46838 => /172.16.38.5:4242] OPEN 18:33:37.147 INFO [ConnectionManager.handleUpstream] - [id: 0xdc462258, /172.16.97.0:36612 => /172.16.38.5:4242] OPEN 18:33:37.413 INFO [ConnectionManager.handleUpstream] - [id: 0x47399539, /172.16.97.0:10846 => /172.16.38.5:4242] OPEN 18:33:37.758 INFO [ConnectionManager.handleUpstream] - [id: 0xeda21e69, /172.16.97.0:36624 => /172.16.38.5:4242] OPEN 18:33:38.056 INFO [ConnectionManager.handleUpstream] - [id: 0x1958a2a7, /172.16.97.0:43876 => /172.16.38.5:4242] OPEN 18:33:38.135 INFO [ConnectionManager.handleUpstream] - [id: 0xc51be6af, /172.16.97.0:43880 => /172.16.38.5:4242] OPEN 18:33:38.440 INFO [ConnectionManager.handleUpstream] - [id: 0xb6a1b273, /172.16.97.0:10847 => /172.16.38.5:4242] OPEN 18:33:38.441 INFO [ConnectionManager.handleUpstream] - [id: 0x3237fa5d, /172.16.97.0:36638 => /172.16.38.5:4242] OPEN 18:33:38.451 INFO [ConnectionManager.handleUpstream] - [id: 0x75834894, /172.16.97.0:36640 => /172.16.38.5:4242] OPEN 18:33:38.460 INFO [ConnectionManager.handleUpstream] - [id: 0xeed730a1, /172.16.97.0:36644 => /172.16.38.5:4242] OPEN 18:33:38.466 INFO [ConnectionManager.handleUpstream] - [id: 0xd934093b, /172.16.97.0:36646 => /172.16.38.5:4242] OPEN 18:33:38.709 INFO [ConnectionManager.handleUpstream] - [id: 0x69d1b5cc, /172.16.97.0:36648 => /172.16.38.5:4242] OPEN 18:33:38.724 INFO [ConnectionManager.handleUpstream] - [id: 0x5d93a95f, /172.16.97.0:36650 => /172.16.38.5:4242] OPEN 18:33:39.188 INFO [ConnectionManager.handleUpstream] - [id: 0xe5789bae, /172.16.97.0:43890 => /172.16.38.5:4242] OPEN 18:33:39.197 INFO [ConnectionManager.handleUpstream] - [id: 0xd7eef76e, /172.16.97.0:46966 => /172.16.38.5:4242] OPEN 18:33:39.229 INFO [ConnectionManager.handleUpstream] - [id: 0xd6498966, /172.16.97.0:36660 => /172.16.38.5:4242] OPEN 18:33:39.312 INFO [ConnectionManager.handleUpstream] - [id: 0xab3a66d4, /172.16.97.0:46978 => /172.16.38.5:4242] OPEN 18:33:39.584 INFO [ConnectionManager.handleUpstream] - [id: 0x9b4be42a, /172.16.97.0:47010 => /172.16.38.5:4242] OPEN 18:33:39.628 INFO [ConnectionManager.handleUpstream] - [id: 0x5b646d34, /172.16.97.0:47012 => /172.16.38.5:4242] OPEN 18:33:40.743 INFO [ConnectionManager.handleUpstream] - [id: 0xde8a79d7, /172.16.97.0:43902 => /172.16.38.5:4242] OPEN 18:33:41.124 INFO [ConnectionManager.handleUpstream] - [id: 0xc8fe1e89, /172.16.97.0:43916 => /172.16.38.5:4242] OPEN 18:33:41.129 INFO [ConnectionManager.handleUpstream] - [id: 0x34301751, /172.16.97.0:43918 => /172.16.38.5:4242] OPEN 18:33:41.133 INFO [ConnectionManager.handleUpstream] - [id: 0x9f185d88, /172.16.97.0:43920 => /172.16.38.5:4242] OPEN 18:33:41.561 INFO [ConnectionManager.handleUpstream] - [id: 0x588cd7f4, /172.16.97.0:43926 => /172.16.38.5:4242] OPEN 18:33:41.899 INFO [ConnectionManager.handleUpstream] - [id: 0x55dcb7fd, /172.16.97.0:36672 => /172.16.38.5:4242] OPEN 18:33:42.071 INFO [ConnectionManager.handleUpstream] - [id: 0xdeaa309f, /172.16.97.0:43948 => /172.16.38.5:4242] OPEN 18:33:42.934 INFO [ConnectionManager.handleUpstream] - [id: 0x793732f7, /172.16.97.0:43980 => /172.16.38.5:4242] OPEN 18:33:43.305 INFO [ConnectionManager.handleUpstream] - [id: 0x5b9e1030, /172.16.97.0:43990 => /172.16.38.5:4242] OPEN 18:33:43.904 INFO [ConnectionManager.handleUpstream] - [id: 0x0930f0fd, /172.16.97.0:44002 => /172.16.38.5:4242] OPEN 18:33:43.971 INFO [ConnectionManager.handleUpstream] - [id: 0x3cf3b85e, /172.16.97.0:44010 => /172.16.38.5:4242] OPEN 18:33:43.982 INFO [ConnectionManager.handleUpstream] - [id: 0xc0b84ca7, /172.16.97.0:44014 => /172.16.38.5:4242] OPEN 18:33:44.038 INFO [ConnectionManager.handleUpstream] - [id: 0x71117d54, /172.16.97.0:44016 => /172.16.38.5:4242] OPEN 18:33:44.055 INFO [ConnectionManager.handleUpstream] - [id: 0xa139cc49, /172.16.97.0:44022 => /172.16.38.5:4242] OPEN 18:33:44.103 INFO [ConnectionManager.handleUpstream] - [id: 0xd93c3780, /172.16.97.0:10848 => /172.16.38.5:4242] OPEN 18:33:44.483 INFO [ConnectionManager.handleUpstream] - [id: 0x3abcc79a, /172.16.97.0:44034 => /172.16.38.5:4242] OPEN 18:33:44.782 INFO [ConnectionManager.handleUpstream] - [id: 0x511f326e, /172.16.97.0:47308 => /172.16.38.5:4242] OPEN 18:33:44.975 INFO [ConnectionManager.handleUpstream] - [id: 0x3c57c678, /172.16.97.0:44044 => /172.16.38.5:4242] OPEN 18:33:45.049 INFO [ConnectionManager.handleUpstream] - [id: 0x9c6d94ae, /172.16.97.0:44020 => /172.16.38.5:4242] OPEN 18:33:45.301 INFO [ConnectionManager.handleUpstream] - [id: 0xbe3b1834, /172.16.97.0:41436 => /172.16.38.5:4242] OPEN 18:33:45.371 INFO [ConnectionManager.handleUpstream] - [id: 0x3c90f00a, /172.16.97.0:47326 => /172.16.38.5:4242] OPEN 18:33:45.753 INFO [ConnectionManager.handleUpstream] - [id: 0x0fedf518, /172.16.97.0:44048 => /172.16.38.5:4242] OPEN 18:33:46.385 INFO [ConnectionManager.handleUpstream] - [id: 0xdb30aea0, /172.16.97.0:47388 => /172.16.38.5:4242] OPEN 18:33:47.167 INFO [ConnectionManager.handleUpstream] - [id: 0xff0c37fa, /172.16.97.0:10850 => /172.16.38.5:4242] OPEN 18:33:48.472 INFO [ConnectionManager.handleUpstream] - [id: 0x712cb89f, /172.16.97.0:10852 => /172.16.38.5:4242] OPEN 18:33:48.476 INFO [ConnectionManager.handleUpstream] - [id: 0x2e137418, /172.16.97.0:36736 => /172.16.38.5:4242] OPEN 18:33:48.732 INFO [ConnectionManager.handleUpstream] - [id: 0x7af0c7af, /172.16.97.0:36740 => /172.16.38.5:4242] OPEN 18:33:48.746 INFO [ConnectionManager.handleUpstream] - [id: 0xe7776b9c, /172.16.97.0:36742 => /172.16.38.5:4242] OPEN 18:33:50.666 INFO [ConnectionManager.handleUpstream] - [id: 0xe1cfb84c, /172.16.97.0:44082 => /172.16.38.5:4242] OPEN 18:33:51.663 INFO [ConnectionManager.handleUpstream] - [id: 0xc723c56a, /172.16.97.0:10855 => /172.16.38.5:4242] OPEN 18:33:51.912 INFO [ConnectionManager.handleUpstream] - [id: 0xf57196a8, /172.16.97.0:36758 => /172.16.38.5:4242] OPEN 18:33:51.963 INFO [ConnectionManager.handleUpstream] - [id: 0x4da5e535, /172.16.97.0:44136 => /172.16.38.5:4242] OPEN 18:33:52.079 INFO [ConnectionManager.handleUpstream] - [id: 0xeb2373c4, /172.16.97.0:44138 => /172.16.38.5:4242] OPEN