Investigate getting rid of custom SCCs

[razvan]

Description

Currently, our OLM packages deploy a custom SecurityContextConstraint based on the hostmount-anyuid SCC. This issue is about getting rid of that custom SCC and move to the nonroot-v2 SCC.

Background / Research

Using the SecurityContext.RunAsUser functionality users can, in theory, set the user that an image is being run as to an arbitrary uid. Our operators currently (as of 2024-07-16) hardcode this to 1000:

pub const NIFI_UID: i64 = 1000;
...
PodSecurityContextBuilder::new()
  .run_as_user(NIFI_UID)
  .run_as_group(0)
  .fs_group(1000)
  .build()

For this to work (read: to be allowed) at least on OpenShift we need to set a SecurityContextConstraint that allows us to run as an arbitrary (non-root) UID.

In the past we deployed our own SCC derived from the hostmount-anyuid (this looked different back then) SCC because the default SCCs didn't allow ephemeral volumes. This has changed in September 2022 without us noticing and all default SCCs now include the ephemeral permission. The hostmount-anyuid SCC allows (as the name implies) Pods to run as any user including root. This is not good and we can probably (we ran some tests and it looks good) switch to nonroot-v2 which is slightly less strict than restricted-v2.

[!NOTE]
nonroot-v2 provides all features of the restricted-v2 SCC, but allows users to run with any non-root UID.

This is a change we can do immediately and should not require any code changes, it should only require changes in the bundling of our OLM packages and it needs to be documented/in the release notes.

Our final goal should be to move to restricted-v2 wherever possible but that is out-of-scope for this issue and will be part of a follow-up.

Value

• Getting rid of the custom SCC will make our life certifying our operators for OpenShift easier as there is one less moving part and we can use defaults
• Getting rid of our old custom SCC will make our defaults more secure for users as Pods cannot be run as root anymore

Dependencies

• This will require OpenShift clusters to test changes
• To setup an OKD cluster on Replicated for deploying OLM bundles, see https://app.nuclino.com/Stackable/Engineering/Set-up-for-deploying-bundles-install-pipeline-operator-ec67d5b5-2941-41a6-8cb8-423d6dee7518

Tasks

• [x] Switch OLM packages to default to the nonroot-v2 SCC (this will probably require changes to the ServiceAccount as well)
• [x] Test all operators on OpenShift using a non-admin user against the new SCC (see results below)
• [x] Document any migration needed (not sure if there is any): No migration is needed. The new SCCs will just take effect with the new SDP version (24.7.0).
• [x] Prepare OLM bundles for certification of release-24.7: see branch stackable-24.7-prep-temp

Acceptance Criteria

• Our operators do not require a custom SCC anymore on OpenShift
• Ideally all operators work with the nonroot-v2 SCC but another less restrictive SCC would also work if there are good reasons for it
  • Moving to the restricted-v2 SCC is out of scope for this issue as we know there is more work to do before we can achieve that

(Information Security) Risk Assessment

This will improve our security of our product as it will allow us to run on default settings of OpenShift without our customers having to audit a custom SCC. It will also allow us to move to a more restrictive SCC going forward.

Quality

We need to run all integration tests (and maybe even demos) across all our products.

Release Notes

Historically our Operator Lifecycle Manager packaging (OLM) for OpenShift would deploy a custom SecurityContextConstraint. This used to be required for us to be able to create ephemeral volumes. All default SCCs in OpenShift in all our supported OpenShift versions now allow this by default. This allows us to switch to use the nonroot-v2 SCC by default.

[adwk67]

Manifests created, operator deployed (🟠) and Openshift test suite run (🟢) for the following operators with no custom SCC:

• 🟢 Commons
• 🟢 Secret (requires stackable-secret-operator-scc)
• 🟢 Listener (requires stackable-listener-operator-scc)
• 🟢 Airflow
• 🟢 Spark-k8s
• 🟢 ZooKeeper
• 🟢 HDFS (script misses clusterrole-nodes: has been added). Chaos-monkey step fails in the kerberos test, due to a missing permision for the listener. Fixed by including a custom scc in the manifests and adding listeners/delete permission.
• 🟢 OPA
• 🟢 Hive
• 🟢 Kafka: expected: 3.7.1-stackable0.0.0-dev != actual: 3.7.1-stackable24.7.0 in upgrade_zookeeper-3.9.2_upgrade_old-3.6.1_upgrade_new-3.7.1_use-client-tls-true_use-client-auth-tls-true_openshift-true (works when corrected locally, but this has been fixed in stackable-utils and so should work for future releases)
• 🟢 HBase
• 🟢 Druid
• 🟢 Superset
• 🟢 Trino
• 🟢 Nifi
• 🔴 Hello-world (not previously certified). Hello-world tests also fail when operator installed via helm. Also when using the original SCC. This command is part of the standard pod start-up commands and matches what is in other operators.

• rm -f /stackable/log/_vector/shutdown rm: cannot remove '/stackable/log/_vector/shutdown': Permission denied

[adwk67]

TODOs / unclear

• [x] do we want "stable" or "24.7" for secret/listener in annotations.yaml?
• [x] is stackable.tech/vendor: Stackable everywhere it should be for secret/listener?
• [x] are stackable-secret-operator-scc and stackable-listener-operator-scc needed: does this undermine the solution to this issue? Try privileged for listener and ~nonroot-v2~ privileged for secret @adwk67 - hdfs/secret/listener/zookeeper tests successful
• [x] do we need "replaces" in the build-manifest.py script? (no, only when patching a release)
• [x] should we have "skips" in the CSV? (no, only when patching a release)
• [x] why don't the scripts deliver securitycontextconstraints for the CSV? (because there is rule that links to the clusterrole, where the scc is mentioned?). The operator has to be able to bind te clusterrole but does not need to use it itself.
• [x] (new issue: https://github.com/stackabletech/listener-operator/pull/216) do we need to propagate all changes in stackable-listener-operator.v24.7.0.clusterserviceversion.yaml back to the helm templates? (e.g. an extra "delete"). Yes, then we stay consistent. @razvan
• [x] https://github.com/stackabletech/issues/issues/624
• [x] Set-up ARO instance for testing/certification @razvan
• [x] we should certify 24.7.0-1 (which can be superceded by 24.7.0)? - No, OLM doesn't care about the names as long as a replaces property is used in the 24.7 channel.

[adwk67]

stackable-utils changes for this issue: https://github.com/stackabletech/stackable-utils/pull/86

[razvan]

This issue addresses the security context part described here.

This issue can now be considered done.

[sbernauer]

This issue can now be considered done.

@razvan can we move this to "Development Done"? I'm not sure what to review here to be honest 🙈

[adwk67]

Yes, closing this as outstanding work will be addressed in separate tickets.

[PaulienVa]

@adwk67 will this also be fixed for the airflow-operator? We also need it there/

[adwk67]

@PaulienVa yes, this will be done for all operators: please follow this issue for updates/progress etc.