Is an OB watch needed (orphaned OBs)?

[jeffvance]

The current design has the potential for orphaned OBs. Per the original design doc:

#### OB Watches (post phase-0)
There is a single, central controller, separate from provisioners, that watches all OBs.
The main (only?) purpose of an OB watch is to detect and handle orphaned OBs -- see above.
The OB controller runs separately from provisioners and thus requires an extra setup step.

The OB watch performs the following:
+ detects a new OB:
  + ensures a matching OBC exists
    + if yes, sets status to "pending"
    + if no, this OB is orphaned so delete it and associated Secet and ConfigMap
+ detects OB update events:
  + if status == "released" then delete OB (and ConfigMap and/or Secret if needed)
+ detects OB delete events (but deletes events can be lost)
  + if the associated OBC is present the OB's Status is set to "released" (the OBC is orphaned but **not** deleted)
  + if the associated Secret and/or ConfigMap are present they are deleted
+ periodically syncs all OBs:
  + fetch all OBs
  + add OBs to the work queue so that an Add event is triggered

and

#### Orphaned Object Buckets (post phase-0)

The implementation of Kubernetes _informers_ recognizes that sometimes Delete events are missed.
Controllers (and this lib), therefore, need to be robust enough to infer deletes via other mechanisms, such as Status or existence of a binding related property. For example, an OBC can be deleted from the cluster, but the delete event is missed and thus the controller doesn't know the delete occurred. In this scenario, the associated OB (and possibly the Secret and/or ConfigMap) remain, resulting in an _orphaned_ OB.

An orphaned OB is an OB with no matching OBC, and thus the state of the cluster is inconsistent.
**Note:** this definition assumes no static OBs for brownfield use.

The solution is to delete orphaned OBs, but how are they detected since OBC watches are triggered by OBC events and there are no "events" associated with orphaned OBs. A solution is to run a centralized, provisioner-agnostic OB controller that watches all OBs, and detects and deletes orphans.
**Note:** this section is subject to change as we consider brownfield bucket implementation details.

There will be only one OB controller running per cluster, not the N controllers needed to support N bucket provisioners. There is still no event to trigger the detection of orphaned OBs, so a reasonably short sync period will cause the OB controller to re-fetch all OBs and look for orphans. When an orphaned OB is found the controller will check to see if the Secret and ConfigMap exist, and if so they will also be deleted.

Even though there are no watches on Secrets and ConfigMaps, there is a lower (no?) probability for orphaned Secrets and/or ConfigMaps since they are deleted prior to the OB, and orphan OB cleanup also delete Secrets and ConfigMaps.

Furthermore, to reduce the chances of an admin deleting a _bound_ OB, and/or Secret and/or ConfigMap, a finalizer is added to these resources. The library's OBC controller will remove the finalizers when an OBC is deleted.
Note: if an orphaned OB must be deleted manually, the finalizer must first be removed manually.

[copejon]

Once an orphaned OB is found, do we also want to delete/revoke the back backing bucket? If the answer is yes, then we may not be able to implement this as a centralized controller. The delete/revoke methods would be unique to individual provisioners.

Following k8s pv controller, the central controller may only Update the OB phase to "released". Then the appropriate provisioner will respond to the update event by checking the phase and if marked "released", carries out a delete/revoke.

[jeffvance]

Note: per issue #131, having the lib add a finalizer to the OBC may be the simplest and least error prone way to protect against orphaned OBs.