Improve Warden Operator for GitHub API calls

[IanKWatts]

Describe the issue The Warden Operator makes a lot of GitHub API calls and now often hits the API rate limit. We can improve the efficiency and performance of the operator by reducing redundant API calls, doing sanity checks on GitHub IDs before trying to add them for repo access, and creating new GitHub users to spread out the load.

What is the Value/Impact? Reduce errors and improve the effectiveness of the operator

What is the plan? How will this get completed?

• Create new GitHub users
• Update warden operator secret for new access tokens
• Update operator code to reduce API calls
• Check for redundant entries in the GitOpsTeam resource. For example, a user that is listed in both gitOpsMembers.readers and gitOpsMembers.writers creates a situation where the operator is constantly updating the user from one role to the other, generating a great deal of unnecessary traffic. See Emerald bfc5f3 GitOpsTeam.

Identify any dependencies n/a

Definition of done

• [x] Operator updated and performance improved

[IanKWatts]

The main problem here, which was that the GitHub APIs were being called so often that we regularly hit the API rate limit, thus breaking the operator, was caused by the way the operator managed the GitOpsTeam resources. A poorly configured GitOpsTeam would result in the operator constantly trying to achieve an impossible state and it would repeatedly make API calls, never getting the desired state to match the actual state.
The operator would fetch the actual state from GitHub and put that in the 'status' of the GitOpsTeam.

• GitHub IDs are case sensitive, but the APIs are not, so an ID can be added using a different case, but the status from GitHub would be reported using the actual ID case and the operator's comparison of the two would fail.
• If a user ID were added to two different roles, such as 'writers' and 'readers', the operator would constantly switch it back and forth, because of course a user ID can only be assigned to one role.
• If an invalid user ID were entered in the GitOpsTeam, it couldn't be added to the repo and so would never achieve the desired state.
• If a user were added that was not already a member of the bcgov-c org, an invitation would be sent and they would be listed as 'pending invitation'. If the invitation were never accepted, the repo would never achieve the desired state.
• If a team ID were entered instead of a user ID, it would fail to be added.
• Sometimes user IDs were entered incorrectly with "@github" at the end, as is used in the 'projectMembers' section, which is for Keycloak configuration.

Because these are not fatal errors, if a GitOpsTeam is not configured properly, the operator will ignore these differences. If users have an access issue, they should review their GitOpsTeam and ensure that all user IDs are entered correctly. Ultimately, the operator was changed to set the status of the resource to the desired state so that it will match and avoid endless reconciliation. For users entered into two roles, the operator checks for duplicates and uses the role with the highest permission. Should probably check the way the operator handles GitOpsAlliances, too, to ensure that those are handled in the same way. I'll create another ticket for that. Also, leader election settings were increased, due to frequent pod restarts in Emerald and sometimes other clusters. Durations are now 4 times the defaults and this seems to have resolved those errors. API calls and operator activity are back to what they should be - that is, very little activity.