Ensure ReplicatedSecret targets are cleaned up in all cases

[Miles-Garnsey]

What is missing?

There are certain cases where a ReplicatedSecret will fail to clean up it's target secrets. This occurs in the following cases:

1. When a source secret is deleted but the ReplicatedSecret which picked it up initially (which happens via labels) still exists. In this case the target will stick around.
2. When a source secret undergoes label changes that move it out of the scope of what is picked up by a ReplicatedSecret that formerly had matching matchLabels, the target secret will not be deleted despite falling out of the scope of the ReplicatedSecret's matchers.

Why do we need it?

Avoiding orphaned resources. Also potentially for security reasons given we use ReplicatedSecrets to control distribution of keys and certs to targeted datacenters. We do not want those resources accessible from namespaces which don't explicitly need them.

┆Issue is synchronized with this Jira Story by Unito

[Miles-Garnsey]

As part of #1267 we are adding a lot more computationally intensive stuff to the cleanup process. In my view that makes this ticket higher priority.

I'm going to suggest (without having investigated this idea properly) that we likely want to move to a model where the ReplicatedSecret keeps track (perhaps in it's status) of what secrets it is replicating. This is important, because otherwise there is a lot of calculation work that needs to be repeated on each reconciliation run.

We should try to move a lot of this stuff out of the controller-side cache and persist it inside the resources themselves, to avoid constant re-calculation.

If feasible, we may also wish to consider including the information about which ReplicatedSecrets are watching a given secret in the origin secret itself, since it is hard to keep track of this too.