reduce concurrent reconciles of the same virtual resource

lizardruss commented 3 weeks ago

What issue type does this pull request address? (keep at least one, remove the others) /kind bugfix

What does this pull request do? Which issues does it resolve? (use resolves #<issue_number> if possible) resolves ENG-4924

Please provide a short message that should be published in the vcluster release notes Fixed an issue where vcluster incorrectly re-created pods during garbage collection

What else do we need to know? The previous method of passing through the syncEventType would potentially enqueue the same object 4 different ways:

virtual object: ns/obj
delete virtual object: delete#ns/obj
physical object: host#ns/obj
delete physical object: delete#host#ns/obj

Usually this would not be a concern, but the following scenario occurs frequently enough to cause a conformance test to fail:

virtual object is deleted
a update event for the virtual object is received that adds the deletionTimestamp
a delete event for the virtual object is received
these events would enqueue the object as objects with the same name in different namespaces from the controller runtime's perspective
the two objects are reconciled concurrently by the controller-runtime
while the Reconcile function is processing the virtual object update the virtual object is deleted, this leads to the SyncToVirtual implementation incorrectly re-creating the virtual object

This PR attempts to solve this by:

Reducing the amount of concurrent reconciles for the same virtual object by dropping the delete# prefix
Using a different method of passing the syncEventType through to Reconcile (and the various sync event types). Event types are recorded by virtual object UID, only the last event is saved, and a delete event for a UID cannot be overwritten.