Fix infinite deletion loops and lock ups

[copejon]

This PR address a couple OBC deletion related issues.

1. When an OBC is deleted before a finalizer can be placed on it, the IsNotFound error is now ignored. The assumption is safe because, if the finalized had been set, then the OBC would not have vanished and thus the OBC would have been found. Once the finalizer is place, it is only removed once all resources are cleaned up. Thus, it is only possible for OBC to vanish in this small window if it was never processed to begin with.

2. When resources cannot be found for deletion, IsNotFound errors are ignored. In cases where a secret, cm, or ob are not found, continue on with deleting the rest of the resources and do not report the errors.

Signed-off-by: jcope jcope@redhat.com

[copejon]

Tested the changes against rook-ceph and aws-s3-provisioner by 1) running create/delete obc calls 1000 times back to back, over multiple iterations.

for n in {1..1000}; do kubectl create -f <obc_file>; kubectl delete obc --all; done

2) forcing a non-nil error return to the Provision() implementation in both provisioners, and rerunning 1).

Prior to this fix, I would expect 1 of 2 cases to occur: a) the OBC would vanish but the library would have continued to try to process the claim, resulting in infinite retry looping b) the CLI would hang indefinitely as a result of the library not properly responding to the deletionTimestamp.

Both of these cases are resolved.
Deleting the OBC before provisioning is complete triggers a follow-on Delete() sync.
Deleting the OBC during an error back off loop now responds appropriately by cleaning up library artifacts and ending the loop.

[jeffvance]

LGTM