I can confirm that multiple operators are letting CR with finalizers untouched and therefor they never get deleted. With that the namespaces are not getting delete as well and possibly some owned resource are also not being deleted. With that behavior clusters fail to run opcap correctly to the end after multiple of those begin to accumulate. There are two problems here:
1) The order the cleanup happens is unclear to me. Looks like the stack is not doing the cleanup in reverse order. That could allow some operators to catch the finalizers and finish the CR.
2) Some operators may actually not be dealing with the finalizer and letting them there.
On the first shot I would do a double check on the operand cleanup by verifying the existence of a finalizer and deleting it anyway. On another issue we may try to write a test to check which operators are actually deleting their finalizers and working correctly on that subject.
I can confirm that multiple operators are letting CR with finalizers untouched and therefor they never get deleted. With that the namespaces are not getting delete as well and possibly some owned resource are also not being deleted. With that behavior clusters fail to run opcap correctly to the end after multiple of those begin to accumulate. There are two problems here:
1) The order the cleanup happens is unclear to me. Looks like the stack is not doing the cleanup in reverse order. That could allow some operators to catch the finalizers and finish the CR.
2) Some operators may actually not be dealing with the finalizer and letting them there.
On the first shot I would do a double check on the operand cleanup by verifying the existence of a finalizer and deleting it anyway. On another issue we may try to write a test to check which operators are actually deleting their finalizers and working correctly on that subject.