`statefulset.{podS,s}ecurityContext` do not work as expected

[chrisseto]

The statefulset.podSecurityContext and statefulset.securityContext fields have not work as one would expect for quite sometime. In the conversion to go, we preserved this behavior (and introduced other issues #1413).

The broken semantics are (roughly): podSecurityContext and securityContext are intermixed and partially applied (FSGroup, FSChangePolicy, RunAsUser, RunAsGroup, AllowPrivilegeEscalation, RunAsNonRoot) to all SecurityContexts and some PodSecurityContexts.

This is clearly less than ideal as:

• Users can't set all fields
• It's unclear which fields propagate where
• It may be surprising that values under statefulset are magically propagated everywhere.
• PodSecurityContext's (eg on Jobs) can't be controlled.

This has left us in a bit of a bind as the most ideal field names are taken.

I'm currently leaning towards:

• add securityContext and podSecurityContext to the root values with no defaults.
• add explicit fields the remaining security contexts following the statefulset.template pattern. e.g. post_install_job.template.securityContext

If any of these fields are provided, statefulset.podSecurityContext and statefulset.securityContext will be ignored. Then in a few releases, we'll remove these fields entirely.

JIRA Link: K8S-309

[chrisseto]

A large chunk of this has been fixed in https://github.com/redpanda-data/helm-charts/pull/1477 by deprecating these fields in favor of a the podTemplate fields.

We'll need to add support for setting the security context(s) of init containers and side cars before full closing this out.