how do you decide which object to remove? for example in the given image there is cat and also remote getting detected.

[woctezuma]

I would guess the detected labels are used. See the numbers for cats and remotes on the GIF.