Fix rect.dendrogram

[talgalili]

See: https://stackoverflow.com/questions/78136216/strange-behaviour-with-rect-dendrogram-in-dendextend-package

[kalaivaniraju]

Apart from two rectangles being drawn, the correct number of rectangles / clusters is also sometimes erroneous.

For example, using the same code below, depending on the hclust.out and h_cut, we get erroneous clusters (4, instead of the correct 3) or double rectangle clusters. For reference of correct clustering, please see the reference dotted vertical line segment drawn at the same height.

For reference, the plots were generated by setting cut_level to 0.5 and 0.7 in the same hclust.out.

hclust.out <- hclust(distance_matrix, method = clustering_method)
h_cut <- cut_level * max(hclust.out$height)

png("example_dend.png")
par(mar = c(5, 1, 1, 10), xpd = TRUE)
dend2 <- as.dendrogram(hclust.out)
dend2 %>% plot(horiz = TRUE, bty = "L")
dend2 %>% rect.dendrogram(h = h_cut, lty = 5, lwd = 0, col = rgb(0.8, 0.8, 0.8, 0.25), horiz = TRUE)
segments(x0 = h_cut, y0 = par("usr")[4], x1 = h_cut, y1 = 0, lty = 2)
dev.off()

[talgalili]

Thanks @kalaivaniraju If you want to propose a PR I'd be happy to review it.