Limit corner radius size to the smallest box dimension

[ErikReider]

One example would be the tiling indicator sometimes experiencing a "sharp" edge when the radius is larger than the height/width of the box.