search

OpenTSDB / opentsdb

A scalable, distributed Time Series Database.
http://opentsdb.net
GNU Lesser General Public License v2.1
4.99k stars 1.25k forks source link

OpenTSDB import fails with PleaseThrottleException #867

Open HariSekhon opened 7 years ago

HariSekhon commented 7 years ago

Hi,

When importing data to HBase 1.1.2 on Hortonworks HDP 2.3 I get the following exception as soon as HBase goes to region split:

org.hbase.async.PleaseThrottleException: 10000 RPCs waiting on "MyTable,,<region>." to come back online
...
Caused by: org.hbase.async.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: MyTable,,<region>. is closing
....

I believe the correct behaviour is to catch the PleaseThrottleException + do an exponential backoff.

Is this something that can be added to the OpenTSDB import tool?

Thanks

Hari

manolama commented 7 years ago

Yeah, this something we need to add, re the backoff. It does mean that HBase is just backed up so if you can pre-split your regions that should help importing.

HariSekhon commented 7 years ago

Yes we did start doing pre-splits as a short term workaround as well as for performance, but I think this is a generic enough issue that the correct handling would significantly improve the robustness of the opentsdb loader.