chore(policy): polish logging

[olix0r]

There are a few things about the policy controller logging that can be cleaned up for consistency and clarity:

• We frequently log ERROR messages when processing resources with unexpected values. These messages are more appropriately emitted at WARN--we want to surface these situations, but they are not really exceptional.
• The leadership status of the status controller is not logged at INFO level, so it's not possible to know about status changes without DEBUG logging.
• We generally use Sentence-cased log messages when emitting user-facing messages. There are a few situations where we are not consistent.
• The status controller reconciliation logging is somewhat noisy and misleading.

DEBUG status::Index: linkerd_policy_controller_k8s_status::index: Lease holder has changed
DEBUG status::Index: linkerd_policy_controller_k8s_status::index: Lease holder reconciling cluster index.name=linkerd-destination-74d7fdc45d-xfb8l
DEBUG status::Index: linkerd_policy_controller_k8s_status::index: Lease holder reconciling cluster index.name=linkerd-destination-74d7fdc45d-xfb8l
DEBUG status::Index: linkerd_policy_controller_k8s_status::index: Lease holder reconciling cluster index.name=linkerd-destination-74d7fdc45d-xfb8l
DEBUG status::Index: linkerd_policy_controller_k8s_status::index: Lease holder has changed
DEBUG status::Index: linkerd_policy_controller_k8s_status::index: Lease holder reconciling cluster index.name=linkerd-destination-74d7fdc45d-xfb8l
DEBUG status::Index: linkerd_policy_controller_k8s_status::index: Lease holder reconciling cluster index.name=linkerd-destination-74d7fdc45d-xfb8l
DEBUG status::Index: linkerd_policy_controller_k8s_status::index: Lease holder reconciling cluster index.name=linkerd-destination-74d7fdc45d-xfb8l
DEBUG status::Index: linkerd_policy_controller_k8s_status::index: Lease holder has changed
DEBUG status::Index: linkerd_policy_controller_k8s_status::index: Lease holder reconciling cluster index.name=linkerd-destination-74d7fdc45d-xfb8l
DEBUG status::Index: linkerd_policy_controller_k8s_status::index: Lease holder reconciling cluster index.name=linkerd-destination-74d7fdc45d-xfb8l
DEBUG status::Index: linkerd_policy_controller_k8s_status::index: Lease holder reconciling cluster index.name=linkerd-destination-74d7fdc45d-xfb8l

The "Lease holder has changed" message actually indicates that the lease has changed, though the holder may be unchanged.

To improve logging clarity, this change does the following:

• Adds an INFO level log when the leadership status of the controller changes.
• Adds an INFO level log when the status controller patches resources.
• Reconciliation logging is moved to TRACE level.
• Consistently uses sentence capitalization in user-facing log messages
• Reduces ERROR messages to WARN when handling invalid user-provided data (including cluster resources). This ensures that ERRORs are reserved for exceptional policy controller states.

[olix0r]

Reconciliation logs have been improved so that we log the number of resources being inspected, as well as the total patches in a reconciliation. Tracing contexts are set so we know which resource is being updated.

The awkward "Lease non-holder skipping controller update" DEBUG messages have been consolidated in a helper as

    fn reconcile_if_leader(&self) {
        let lease = self.claims.borrow();
        if !lease.is_current_for(&self.name) {
            tracing::trace!(%lease.holder, "Reconcilation skipped");
            return;
        }
        self.reconcile();
    } 

Example log output here.

[cratelyn]

note that two ci jobs appear to be failing, my understanding is that these are known to be flaky.