Custom Rowkey Design in OpenTSDB

[bhaukadam007]

Through OpenTSDB 2.2.0 I am trying to insert data in HBase but I am facing HotSpotting issue while inserting data. Is there any way to design custom rowkey for OpenTSDB?

[RestfulBlue]

did you try to use salting?

[bhaukadam007]

Actually currently data present in HBase is in TBs and we inserted that data in HBase through OpenTSDB without using salting.So if we use salting to avoid hotspotting then we can lost our previously inserted data.Can you provide any help or reference how to design custom row key in OpenTSDB .

[manolama]

We have the same issue and will address this in 3.0 wherin you can start a new table and query across both the old and new. And we're exploring a new schema as well.

[bhaukadam007]

Currently, we are using IMPORT(_http://opentsdb.net/docs/build/html/user_guide/cli/import.html_) command to bulk insert data in OpenTSDB, we haven't written any JAVA code to insert data.

As 3.0 version is not released then how can we fix this problem of hotspotting?

Actually earlier we were inserting data like :

metric_name timestamp value tag1=value1

but now we have added one more tag in our data and that has caused HBase Hotspotting, now our data looks like :

metric_name timestamp value tag1=value1 tag2=value2

Do you think adding one more tag can cause Hotspotting? If yes then How can we fix this?

[bhaukadam007]

In the above comment, I have mentioned how we were inserting data in HBase via OpenTSDB(2.2.0) and now we have made changes in our data, we simply added one more tag and we struck in HBase hotspotting issue. I am expecting some quick & concrete resolution as we have to take this changes in the PRODUCTION environment. Please take this issue on high priority. We have around 3+ Terabyte data in our Production environment if we have to migrate our data from OpenTSDB 2.2.0 to another version then how will we do this as we can't put our Prod data on risk.

Please let me know if you need any further information from our side. If this issue is not fixed at earliest then we have to find an option against OpenTSDB.

[bhaukadam007]

As you mentioned earlier that Hotspotting issue will be fixed in OpenTSDB version 3.0 .But OpenTSDB Version 3.0 is not released yet.So when we expect the OpenTSDB version 3.0 will be released?

[bhaukadam007]

We have already mentioned above about data we are trying to insert in HBase using OpenTSDB, but unfortunately we encountered with HBase Hotspotting issue. Earlier you have mentioned that you gonna fix this issue in v3.0. We have checked URL(http://opentsdb.net/docs/build/html/new.html) but didn't found anything about this issue. As I already mentioned that we have to take away this changes to Production, so any updates from your side on this is appreciable

[johann8384]

Pre-Splitting the table, or Salting are the only ways to avoid this issue. OpenTSDB 3.0 is still months away.

If you need assistance with inplementing Salting or Pre-Splitting send a note on the mailing list and myself or others will try to help you there.

Thank you!

[johann8384]

Additionally, you may want to use a second TSD to read and re-insert historical data into a salted table to help facilitate a cutover.