Mutate and Validate RayCluster on SecurityContext

[ChristianZaccaria]

Issue link

Jira: https://issues.redhat.com/browse/RHOAIENG-7638

What changes have been made

• The Mutating Webhook will now add the required SecurityContext in the RayCluster's head and worker containers.
• The Validating Webhook will now check for the required SecurityContext in the RayCluster's head and worker containers.

Verification steps

1. Create Data science project in RHOAI
2. Create and start workbench
3. Find OpenShift namespace corresponding to the Data science project name, add labels to the namespace resource with value:
   • pod-security.kubernetes.io/enforce: restricted
   • pod-security.kubernetes.io/enforce-version: v1.24
4. Clone SDK demo notebooks in Workbench
5. Setup Kueue resources
6. Run i.e. 0_basic_ray.ipynb Notebook, try to create Ray cluster
7. RayCluster should start with or without the labels set in the Namespace resource.
8. You may also check that the SecurityContext is added to both head and worker containers when the mentioned labels have been set.
9. Attempt to create a RayCluster without the SecurityContext, and attempt to update the SecurityContext once the RayCluster is created. The ValidatingWebhook should reject these actions.

Checks

• [ ] I've made sure the tests are passing.
• Testing Strategy
  • [ ] Unit tests
  • [ ] Manual tests
  • [ ] Testing is not required for this change

[openshift-ci[bot]]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Once this PR has been reviewed and has the lgtm label, please ask for approval from christianzaccaria. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files: - **[OWNERS](https://github.com/project-codeflare/codeflare-operator/blob/main/OWNERS)** Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

[Fiona-Waters]

I've run this on my cluster:

• I'm logged in as an admin user
• I have not set the labels in the namespace
• I can submit a raycluster via the SDK and the webhook successfully adds the security context in each container of ray head and worker pods
• I tried to edit the yaml to update or remove the security context in the raycluster CR and was stopped from doing so.
• I can submit a job but it is in pending state (ray dashboard) for a few minutes and then fails.

• I get this error (using cluster_job_client demo notebook)

  'Traceback (most recent call last):\n File "/tmp/ray/session_2024-06-21_04-50-03_677463_8/runtime_resources/working_dir_files/_ray_pkg_262c3c69a1596acc/mnist_fashion.py", line 84, in \n results = trainer.fit()\n File "/home/ray/anaconda3/lib/python3.9/site-packages/ray/train/base_trainer.py", line 633, in fit\n tuner = Tuner(\n File "/home/ray/anaconda3/lib/python3.9/site-packages/ray/tune/tuner.py", line 179, in init\n self._local_tuner = TunerInternal(**kwargs)\n File "/home/ray/anaconda3/lib/python3.9/site-packages/ray/tune/impl/tuner_internal.py", line 144, in init\n ) = self.setup_create_experiment_checkpoint_dir(\n File "/home/ray/anaconda3/lib/python3.9/site-packages/ray/tune/impl/tuner_internal.py", line 522, in setup_create_experiment_checkpoint_dir\n os.makedirs(experiment_path, exist_ok=True)\n File "/home/ray/anaconda3/lib/python3.9/os.py", line 215, in makedirs\n makedirs(head, exist_ok=exist_ok)\n File "/home/ray/anaconda3/lib/python3.9/os.py", line 225, in makedirs\n mkdir(name, mode)\nPermissionError: [Errno 13] Permission denied: \'/home/ray/ray_results\'\n'

[KPostOffice]

@ChristianZaccaria Should we close this then in lieu of https://github.com/opendatahub-io/kuberay/pull/19 ?

[ChristianZaccaria]

Closing in lieu of https://github.com/opendatahub-io/kuberay/pull/19