K8SPSMDB-1132: add `spec.secrets.keyFile` field

[pooknull]

https://perconadev.atlassian.net/browse/K8SPSMDB-1132

DESCRIPTION

Problem: We can't specify custom keyfile secret in the cr.yaml

Solution: Add field .spec.secrets.keyFile to the cr.yaml

CHECKLIST

Jira

• [x] Is the Jira ticket created and referenced properly?
• [x] Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
• [x] Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

• [x] Is an E2E test/test case added for the new feature/change?
• [ ] Are unit tests added where appropriate?
• [x] Are OpenShift compare files changed for E2E tests (compare/*-oc.yml)?

Config/Logging/Testability

• [x] Are all needed new/changed options added to default YAML files?
• [x] Did we add proper logging messages for operator actions?
• [x] Did we ensure compatibility with the previous version or cluster upgrade process?
• [x] Does the change support oldest and newest supported MongoDB version?
• [x] Does the change support oldest and newest supported Kubernetes version?

[inelpandzic]

@egegunes should operator automatically set

  tls:
    mode: allowTLS

When the user sets "keyFile: test-keyfile" via CR? Because now, if the user specifies this option, the mongod will not use key file. And at the same time I see a problem with mongos right now. When I set "mode: allowTLS" mongos still use x509 for clusterAuthMode and it is a bug that should be fixed as well.

@pooknull don't forget to update helm charts with these new fields.

If you ask me, I wouldn't touch user CR, rather than that we should probably prevent setting keyFile if proper tls.mode is not set.

[egegunes]

@hors I agree with @inelpandzic, operator shouldn't do this automatically but return error if tls.mode is not compatible with keyFile

[egegunes]

@pooknull we need to use keyFile authentication if keyFile secret is specified in cr.yaml, no matter what tls.mode is

[egegunes]

@pooknull we also need to use keyFile auth for mongos if secret is specified

[JNKPercona]

Test name	Status
arbiter	passed
balancer	passed
custom-replset-name	passed
custom-tls	passed
custom-users-roles	passed
custom-users-roles-sharded	passed
cross-site-sharded	passed
data-at-rest-encryption	passed
data-sharded	passed
demand-backup	passed
demand-backup-eks-credentials	passed
demand-backup-physical	passed
demand-backup-physical-sharded	passed
demand-backup-sharded	passed
expose-sharded	passed
ignore-labels-annotations	passed
init-deploy	passed
finalizer	passed
ldap	passed
ldap-tls	passed
limits	passed
liveness	passed
mongod-major-upgrade	passed
mongod-major-upgrade-sharded	passed
monitoring-2-0	failure
multi-cluster-service	passed
non-voting	passed
one-pod	passed
operator-self-healing-chaos	passed
pitr	passed
pitr-sharded	passed
pitr-physical	passed
pvc-resize	passed
recover-no-primary	passed
replset-overrides	passed
rs-shard-migration	passed
scaling	passed
scheduled-backup	passed
security-context	passed
self-healing-chaos	passed
service-per-pod	passed
serviceless-external-nodes	passed
smart-update	passed
split-horizon	passed
storage	passed
tls-issue-cert-manager	passed
upgrade	passed
upgrade-consistency	passed
upgrade-consistency-sharded-tls	passed
upgrade-sharded	passed
users	passed
version-service	passed
We run 52 out of 52

commit: https://github.com/percona/percona-server-mongodb-operator/pull/1639/commits/7728af520613aff326757f0d5b8ca612fe7ac976 image: perconalab/percona-server-mongodb-operator:PR-1639-7728af52