Invalidate only when things really needs to be invalidated

[Arneball]

Otherwise we have a lot of meaningless drawings, messing up battery lifetime and performance