No network separation of user notebooks

Bug Description

Right now in CKF user-a can connect to the url of notebook that is deployed in a Profile that belongs touser-b, where UserA should not have any access.

This is not the case for upstream Kubeflow.

The reason this is the case for CKF is because of this issue we initially encountered in KFP https://github.com/canonical/kfp-operators/issues/355. Specifically, in CKF we now have an extra AuthorizationPolicy that allows any user (any value of kubeflow-userid) header to talk to the workloads of a Profile.

The root cause of this is as explained in https://github.com/canonical/kfp-operators/issues/355#issuecomment-1776809861, that KFP API pod does not have any sidecar. So the default upstream AuthorizationPolicy doesn't allow KFP to talk to pods in user's namespaces. More on this can be found in https://github.com/canonical/kfp-operators/issues/355#issuecomment-2410875457

To Reproduce

Install CKF 1.9
Login and create the default admin profile

Manually apply the following Profile

apiVersion: kubeflow.org/v1
kind: Profile
metadata:
  name: kimonas
spec:
  owner:
    kind: User
    name: kimonas
  resourceQuotaSpec: {}

Deploy the following notebook in the kimonas namespace
Access the URL of the kimonas notebook, while logged in as admin

Environment

CKF 1.9

Relevant Log Output

<none>

Additional Context

A temporary workaround is to update the following AuthorizationPolicy from allowing all requests to just the namespace owner, and not *. https://github.com/canonical/kfp-operators/blob/track/2.2/charms/kfp-profile-controller/files/upstream/sync.py#L480

This will block users to access other notebooks from their browsers.

The following though would still mean that users from their Pods (notebooks) could curl K8s svc urls of other notebooks, explicitly set the kubeflow-userid header, and reach workloads in other user namespaces.

canonical / bundle-kubeflow