I use ganglia to monitor hadoop with hadoop-metrics2. I use metric named "dfs.datanode.HeartbeatsAvgTime" to judge whether the datanode is up or down. When the datanode service works fine, the value of metric "dfs.datanode.HeartbeatsAvgTime" varies between 1s to 4s. But if I stop the datanode service, the value of the metric remain not changed (the value remains at 2.6s ).
After I check the corresponding *.rrd file of the metric, I found that the value of metric is also updated in rrd with the value 2.6s. I thought that after I stoped datanode service, the hadoop will never send metric to ganglia any more, so the value of the metric will become 0 or infinatly or UNKNOWN. But I am wrong.
If I missed any important points about ganglia or hadoop-metrics2 or rrd ,please let me know.Think you.
I use ganglia to monitor hadoop with hadoop-metrics2. I use metric named "dfs.datanode.HeartbeatsAvgTime" to judge whether the datanode is up or down. When the datanode service works fine, the value of metric "dfs.datanode.HeartbeatsAvgTime" varies between 1s to 4s. But if I stop the datanode service, the value of the metric remain not changed (the value remains at 2.6s ).
After I check the corresponding *.rrd file of the metric, I found that the value of metric is also updated in rrd with the value 2.6s. I thought that after I stoped datanode service, the hadoop will never send metric to ganglia any more, so the value of the metric will become 0 or infinatly or UNKNOWN. But I am wrong.
If I missed any important points about ganglia or hadoop-metrics2 or rrd ,please let me know.Think you.
I haved also raised a question in StackOverflow: https://stackoverflow.com/questions/43518412/ganglia-the-graph-in-ganglia-remians-unchanged-after-i-stop-hadoop-datanode .there are some images you can refer to. @vvuksan @plaguedbypenguins