Kubeflow 1.10 Enterprise grade Agenda

[juliusvonkohout]

Validation Checklist

• [X] Is this a Kubeflow issue?
• [X] Are you posting in the right repository ?
• [X] Did you follow the installation guide https://github.com/kubeflow/manifests?tab=readme-ov-file ?
• [X] Is the issue report properly structured and detailed with version numbers?
• [X] Is this for Kubeflow development ?
• [X] Would you like to work on this issue?
• [X] Join our slack channel using wg-manifests.

Version

master

Describe your issue

For general Kubeflow:

• Missing namespace in authorizationpolicy https://github.com/kubeflow/manifests/pull/2298#
• https://github.com/kubeflow/kubeflow/issues/6708
• Renaming Notebooks to workbenches as discussed with Kimonas Sotirchos. There are not only Jupyterlabs but also Rstudios and Vscodes and other customer specific applications (e.g. Labelstudio). If possible make the istio-virtualservice redirect path configurable instead of always using NB_PREFIX.
• Docker -> OCI Update form-image.component.html https://github.com/kubeflow/kubeflow/pull/6673
• The kserve logo does have a different color than the other logos https://github.com/kubeflow/kubeflow/pull/6656
• oidc-authservice https://github.com/kubeflow/manifests/pull/2150 and https://github.com/kubeflow/kubeflow/pull/6609 for proper programatic authentication via tokens and serviceaccounts
• Remember the most recently selected namespace https://github.com/kubeflow/kubeflow/pull/6663/
• There is also another meta issue now https://github.com/kubeflow/kubeflow/issues/6672

Security:

• https://github.com/kubeflow/manifests/pull/2357
• https://github.com/kubeflow/katib/pull/1983.
• https://github.com/kubeflow/kubeflow/issues/6702
• https://github.com/kubeflow/manifests/pull/2304
• Are you open to a rootless istio (istio-cni) for Kubeflow 1.7 ? Currently almost everything runs insecurely as root. I could also add offical Kubernetes podsecuritystandards: restricted then for Kubernetes 1.23+. https://www.arrikto.com/blog/securing-your-kubeflow-deployment-with-kyverno-policies/?utm_content=222104897&utm_medium=social&utm_source=linkedin&hss_channel=lis-ZPwp5HFOSx is also interesting. But they seem to not know how dangerous root containers are. I think for kubeflow PSA /PSS/ Kyverno are the best options.
• Namespace sharing is completely broken from a security perspective. You can easily escalate from default-viewer to default-editor. Furthermore a malicious collaborator can easily extract the serviceaccount token, leave the company and still login with that as bearer token and impersonate other users. Very dangerous. I tried a lot, but the only sane way if found is to add a checkbox on the collaborator page that says "Delete all secrets, serviceaccounts and pods when removing a collborator via kubectl delete serviceaccounts,secrets,pods --all --force. The default secrets and serviceaccounts will be immideately recreated by the Kubeflow profile controllers, but manually created ones are your own resposibility. This might lead to a very short service interuption for e.g. KFP and other Kubeflow components." We could just display this as Information or try to implement it in the profile controller.
• Katib Narrow and explicit RBAC https://github.com/kubeflow/katib/issues/1639
• The pipeline UI allows reading other peoples artifacts. The artifact proxy in the user namespace is insecure and obsolete. In the UI you can just get the artifact link from another user, remove the ?namespace=xxx parameter at the end and the UI server will fake the corresponding user for you. So if you know the S3/GCS path you can read other guys artifacts.

For Pipelines maybe :

• [feature] Seek alternative for object store solution other than MinIO https://github.com/kubeflow/pipelines/issues/7878
• Proper cache configuration for v2 pipelines too https://github.com/kubeflow/pipelines/pull/8270
• Support big (5+ GB) s3 artifacts in pipelines https://github.com/argoproj/argo-workflows/issues/9525 and https://github.com/argoproj/argo-workflows/issues/1322 (they are not solved)
• feat(backend): isolate artifacts per namespace/profile/user using only one bucket https://github.com/kubeflow/pipelines/pull/7725 We need to add gcs and aws s3 support to get it merged
• implement a caching strategy for PVC data_passing_method. (https://kubeflow-pipelines.readthedocs.io/en/latest/_modules/kfp/dsl/_pipeline.html?highlight=data_passing_method#). For example if one uses a pvc named after the pipeline for all executions of that pipeline versions this could be useful.
• https://github.com/kubeflow/pipelines/issues/4790

Good first issues

• listruns performance is bad because of shitty SQL queries https://github.com/kubeflow/pipelines/issues/9890 please use https://github.com/kubeflow/pipelines/pull/9680 as a reference. Here the tale begins https://github.com/kubeflow/pipelines/blob/ad058b5321f5fe74bbd10845778e6627fb9aa5cb/backend/src/apiserver/storage/run_store.go#L109. Maybe the frontend is not using pagination as well. I do not know where the bottleneck is.
• Try out kfp v2 rootless https://github.com/kubeflow/pipelines/pull/7916
• SDK support for v1 namespaced pipelines https://github.com/kubeflow/pipelines/pull/8512
• Reviewing Improve the performance of most Kubeflow backends https://github.com/kubeflow/kubeflow/pull/7080
• Workbenches as successor to notebooks https://github.com/kubeflow/kubeflow/issues/7156
• General culling in old "noteboks" https://github.com/kubeflow/kubeflow/issues/6920
• Support big (5+ GB) s3 artifacts in pipelines https://github.com/argoproj/argo-workflows/issues/9525 and https://github.com/argoproj/argo-workflows/issues/1322 (they are not solved)
• admission webhook performance https://github.com/kubeflow/kubeflow/issues/6708
• Ray istio/headless https://github.com/ray-project/kuberay/issues/1005

Steps to reproduce the issue

Install Kubeflow

Put here any screenshots or videos (optional)

No response

[rimolive]

Can we work on an alternative Container Registry to overcome the Docker Hub rate limit issues?

[juliusvonkohout]

@rimolive Yes, @thesuperzapper wants ghcr if I remember correctly and I also think that GitHub itself as registry might be a good choice.

[andreyvelich]

I think, we should discuss pros and cons to use Docker Hub vs ghcr. Since we can ask CNCF for subscription for Docker Hub account for our official Kubeflow DockerHub: https://hub.docker.com/orgs/kubeflow, it might solve an issue with Docker Hub rate limits. It gives 5000 pulls per day for every user.

[kimwnasptd]

@juliusvonkohout should we also warm up again the discussion about groups support, now that the first step of oauth2-proxy is merged and we are passing id-tokens on requests?

https://github.com/kubeflow/kubeflow/issues/4188 https://github.com/kubeflow/kubeflow/issues/7085

[juliusvonkohout]

@juliusvonkohout should we also warm up again the discussion about groups support, now that the first step of oauth2-proxy is merged and we are passing id-tokens on requests?

kubeflow/kubeflow#4188 kubeflow/kubeflow#7085

Yes please :-) Lets discuss it in the Kubeflow platform/manifests meeting. We really have to think the architecture trough.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[juliusvonkohout]

/lifecycle frozen

[juliusvonkohout]

I merged the remaining issues into https://github.com/kubeflow/manifests/issues/2763