Closed rclaire closed 8 years ago
i've seen weird behavior like this, and our attempt for a workaround was to upgrade to HBase 0.98 and 1.x. and we haven't seen that exception since, but ymmv.
Thanks mxk1235
@rclaire and you didn't enable the RPC timeouts right? There is a potential issue there with 0.94 and timeouts.
@rclaire Try out the next branch now. It's using the 1.7.1-SNAPSHOT of asynchbase with the fix for pre-protobuf RPCs.
Should be due to the AsyncHBase issue. Closing for now, please re-open if it pops up again please.
Hi,
We've been observing frequent hanging of openTSDB during testing of the 2.2.0 RC1 release, in a similar environment to large 2.1.0 production deployment
The most common error message that we have seen in the openTSDB logs is the following:
Testing is done with a staging environment, starting with empty HBase tables. We've been trying multiple combinations of settings (Salting, random UID assignment, uid width, Async IO on and off, etc...), JVM options, and HBase regions splitting, but the problem persist.
The types of things that we are seeing are:
Example of stacktrace during hung startup:
Example of stacktrace during hung query: