Navigator upgrade

[wallrj]

Things to consider, in no particular order:

• [ ] Documentation describing how to upgrade Navigator
• [ ] Add Navigator upgrade tests
• [ ] Controller Deployment should upgradable
• [ ] API server Deployment should be upgradable
• [ ] Helm chart should support Navigator upgrades
• [ ] New Controller should update or replace Kubernetes resources that have changed between versions (e.g. Modifications to statefulsets [#308] , services, rbac roles and bindings).
• [ ] New Pilot images should be rolled out.
• [ ] All with zero or minimal downtime to the databases controlled by Navigator.

/kind feature

[munnerz]

New Controller should update or replace Kubernetes resources that have changed between versions (e.g. Modifications to statefulsets [#308] , services, rbac roles and bindings).

I don't think we should treat an upgrade of navigator as a special case of upgrading the clusters it manages. It should be treated in the same way as any other sync, e.g. if a user were to manually edit a StatefulSet resource managed by Navigator, Navigator should reconcile the changes to that StatefulSet using the same mechanism as is used during an upgrade to navigator-controller (e.g. navigator-controller should observe what the current resource requirements are, and what they should be, and reconcile that).