Ml Pipeline Artifact 503 RBAC:Access Denied

[tugasakhirai21]

I installed Kubeflow on my on-premise Kubernetes cluster, and most things seem to be working fine. However, I'm encountering an issue when trying to access artifacts from the Run dashboard. Whenever I click on any step and try to access the corresponding artifact link on the right side, I receive an "RBAC: access denied" message.

I can log in to Minio directly and access these artifacts manually. Additionally, I noticed that when I remove the namespace component from the URL, I can download the artifact files successfully.

I've tried all solutions mentioned in this forum, but none of them have worked for me so far.

Any insights or suggestions on how to resolve this issue?

Originally posted by @tugasakhirai21 in https://github.com/kubeflow/pipelines/issues/6839#issuecomment-2077130252

[tugasakhirai21]

Below the network when i try to access artifacts tab

When i remove the namespace URL Parameter, HTTP GET response succeeded

Below the authorization policy and rolebinding in the kubeflow and kubeflow-user-example-com namespace

master@k8s-master:$ kubectl get authorizationpolicy -n kubeflow NAME AGE bind-ml-pipeline-nb-kubeflow-user-example-com 25d central-dashboard 27d jupyter-web-app 27d katib-ui 27d kserve-models-web-app 27d metadata-grpc-service 27d minio-service 27d ml-pipeline 27d ml-pipeline-ui 27d ml-pipeline-visualizationserver 27d mysql 27d profiles-kfam 27d service-cache-server 27d tensorboards-web-app 27d volumes-web-app 27d master@k8s-master:$ kubectl get authorizationpolicy -n kubeflow-user-example-com NAME AGE minio-service 21d ml-pipeline-visualizationserver 27d ns-owner-access-istio 27d master@k8s-master:$ kubectl get clusterrole -n kubeflow NAME CREATED AT addressable-resolver 2024-03-30T00:22:21Z admin 2024-03-29T13:25:57Z admission-webhook-cluster-role 2024-03-30T00:22:21Z admission-webhook-kubeflow-poddefaults-admin 2024-03-30T00:22:21Z admission-webhook-kubeflow-poddefaults-edit 2024-03-30T00:22:21Z admission-webhook-kubeflow-poddefaults-view 2024-03-30T00:22:21Z aggregate-to-kubeflow-pipelines-edit 2024-03-30T00:22:21Z aggregate-to-kubeflow-pipelines-view 2024-03-30T00:22:21Z argo-aggregate-to-admin 2024-03-30T00:22:21Z argo-aggregate-to-edit 2024-03-30T00:22:21Z argo-aggregate-to-view 2024-03-30T00:22:21Z argo-cluster-role 2024-03-30T00:22:21Z authn-delegator 2024-03-29T16:24:26Z broker-addressable-resolver 2024-03-30T00:22:21Z builtin-podspecable-binding 2024-03-30T00:22:21Z calico-kube-controllers 2024-03-29T13:29:37Z calico-node 2024-03-29T13:29:37Z centraldashboard 2024-03-30T00:22:21Z cert-manager-cainjector 2024-03-30T00:22:21Z cert-manager-controller-approve:cert-manager-io 2024-03-30T00:22:21Z cert-manager-controller-certificates 2024-03-30T00:22:21Z cert-manager-controller-certificatesigningrequests 2024-03-30T00:22:21Z cert-manager-controller-challenges 2024-03-30T00:22:21Z cert-manager-controller-clusterissuers 2024-03-30T00:22:21Z cert-manager-controller-ingress-shim 2024-03-30T00:22:21Z cert-manager-controller-issuers 2024-03-30T00:22:21Z cert-manager-controller-orders 2024-03-30T00:22:21Z cert-manager-edit 2024-03-30T00:22:21Z cert-manager-view 2024-03-30T00:22:21Z cert-manager-webhook:subjectaccessreviews 2024-03-30T00:22:21Z channel-addressable-resolver 2024-03-30T00:22:21Z channelable-manipulator 2024-03-30T00:22:21Z cluster-admin 2024-03-29T13:25:57Z dex 2024-03-30T00:22:21Z edit 2024-03-29T13:25:57Z eventing-broker-filter 2024-03-30T00:22:21Z eventing-broker-ingress 2024-03-30T00:22:21Z eventing-config-reader 2024-03-30T00:22:21Z eventing-sources-source-observer 2024-03-30T00:22:21Z flows-addressable-resolver 2024-03-30T00:22:21Z gpu-operator 2024-03-29T14:57:08Z gpu-operator-node-feature-discovery 2024-03-29T14:57:08Z gpu-operator-node-feature-discovery-gc 2024-03-29T14:57:08Z istio-reader-clusterrole-istio-system 2024-03-30T00:22:21Z istio-reader-istio-system 2024-03-30T00:22:21Z istiod-clusterrole-istio-system 2024-03-30T00:22:21Z istiod-gateway-controller-istio-system 2024-03-30T00:22:21Z istiod-istio-system 2024-03-30T00:22:21Z jupyter-web-app-cluster-role 2024-03-30T00:22:21Z jupyter-web-app-kubeflow-notebook-ui-admin 2024-03-30T00:22:21Z jupyter-web-app-kubeflow-notebook-ui-edit 2024-03-30T00:22:21Z jupyter-web-app-kubeflow-notebook-ui-view 2024-03-30T00:22:21Z katib-controller 2024-03-30T00:22:21Z katib-ui 2024-03-30T00:22:21Z knative-bindings-namespaced-admin 2024-03-30T00:22:21Z knative-eventing-controller 2024-03-30T00:22:21Z knative-eventing-namespaced-admin 2024-03-30T00:22:21Z knative-eventing-namespaced-edit 2024-03-30T00:22:21Z knative-eventing-namespaced-view 2024-03-30T00:22:21Z knative-eventing-pingsource-mt-adapter 2024-03-30T00:22:21Z knative-eventing-sources-controller 2024-03-30T00:22:21Z knative-eventing-webhook 2024-03-30T00:22:21Z knative-flows-namespaced-admin 2024-03-30T00:22:21Z knative-messaging-namespaced-admin 2024-03-30T00:22:21Z knative-serving-addressable-resolver 2024-03-30T00:22:21Z knative-serving-admin 2024-03-30T00:22:21Z knative-serving-aggregated-addressable-resolver 2024-03-30T00:22:21Z knative-serving-core 2024-03-30T00:22:21Z knative-serving-istio 2024-03-30T00:22:21Z knative-serving-namespaced-admin 2024-03-30T00:22:21Z knative-serving-namespaced-edit 2024-03-30T00:22:21Z knative-serving-namespaced-view 2024-03-30T00:22:21Z knative-serving-podspecable-binding 2024-03-30T00:22:21Z knative-sources-namespaced-admin 2024-03-30T00:22:21Z kserve-manager-role 2024-03-30T00:22:22Z kserve-models-web-app-cluster-role 2024-03-30T00:22:22Z kserve-proxy-role 2024-03-30T00:22:22Z kubeadm:get-nodes 2024-03-29T13:25:59Z kubeflow-admin 2024-03-30T00:22:22Z kubeflow-edit 2024-03-30T00:22:22Z kubeflow-istio-admin 2024-03-30T00:22:22Z kubeflow-istio-edit 2024-03-30T00:22:22Z kubeflow-istio-view 2024-03-30T00:22:22Z kubeflow-katib-admin 2024-03-30T00:22:22Z kubeflow-katib-edit 2024-03-30T00:22:22Z kubeflow-katib-view 2024-03-30T00:22:22Z kubeflow-kserve-admin 2024-03-30T00:22:22Z kubeflow-kserve-edit 2024-03-30T00:22:22Z kubeflow-kserve-view 2024-03-30T00:22:22Z kubeflow-kubernetes-admin 2024-03-30T00:22:22Z kubeflow-kubernetes-edit 2024-03-30T00:22:22Z kubeflow-kubernetes-view 2024-03-30T00:22:22Z kubeflow-pipelines-cache-role 2024-03-30T00:22:22Z kubeflow-pipelines-edit 2024-03-30T00:22:22Z kubeflow-pipelines-metadata-writer-role 2024-03-30T00:22:22Z kubeflow-pipelines-view 2024-03-30T00:22:22Z kubeflow-training-admin 2024-03-30T00:22:22Z kubeflow-training-edit 2024-03-30T00:22:22Z kubeflow-training-view 2024-03-30T00:22:22Z kubeflow-view 2024-03-30T00:22:22Z kubernetes-dashboard-metrics-scraper 2024-04-04T09:57:55Z local-path-provisioner-role 2024-03-29T14:01:51Z meta-channelable-manipulator 2024-03-30T00:22:22Z ml-pipeline 2024-03-30T00:22:22Z ml-pipeline-persistenceagent-role 2024-03-30T00:22:22Z ml-pipeline-scheduledworkflow-role 2024-03-30T00:22:22Z ml-pipeline-ui 2024-03-30T00:22:22Z ml-pipeline-viewer-controller-role 2024-03-30T00:22:22Z notebook-controller-kubeflow-notebooks-admin 2024-03-30T00:22:22Z notebook-controller-kubeflow-notebooks-edit 2024-03-30T00:22:22Z notebook-controller-kubeflow-notebooks-view 2024-03-30T00:22:22Z notebook-controller-role 2024-03-30T00:22:22Z nvidia-device-plugin 2024-03-29T14:57:33Z nvidia-gpu-feature-discovery 2024-03-29T14:57:33Z nvidia-mig-manager 2024-03-29T14:57:33Z nvidia-operator-validator 2024-03-29T14:57:33Z podspecable-binding 2024-03-30T00:22:22Z pvcviewer-metrics-reader 2024-03-30T00:22:22Z pvcviewer-proxy-role 2024-03-30T00:22:22Z pvcviewer-role 2024-03-30T00:22:22Z seldon-manager-role-seldon-system 2024-04-24T13:38:19Z seldon-manager-sas-role-seldon-system 2024-04-24T13:38:19Z seldon-spartakus-volunteer-seldon-system 2024-04-24T13:38:19Z service-addressable-resolver 2024-03-30T00:22:22Z serving-addressable-resolver 2024-03-30T00:22:22Z source-observer 2024-03-30T00:22:22Z system:aggregate-to-admin 2024-03-29T13:25:57Z system:aggregate-to-edit 2024-03-29T13:25:57Z system:aggregate-to-view 2024-03-29T13:25:57Z system:auth-delegator 2024-03-29T13:25:57Z system:basic-user 2024-03-29T13:25:57Z system:certificates.k8s.io:certificatesigningrequests:nodeclient 2024-03-29T13:25:57Z system:certificates.k8s.io:certificatesigningrequests:selfnodeclient 2024-03-29T13:25:57Z system:certificates.k8s.io:kube-apiserver-client-approver 2024-03-29T13:25:57Z system:certificates.k8s.io:kube-apiserver-client-kubelet-approver 2024-03-29T13:25:57Z system:certificates.k8s.io:kubelet-serving-approver 2024-03-29T13:25:57Z system:certificates.k8s.io:legacy-unknown-approver 2024-03-29T13:25:57Z system:controller:attachdetach-controller 2024-03-29T13:25:57Z system:controller:certificate-controller 2024-03-29T13:25:57Z system:controller:clusterrole-aggregation-controller 2024-03-29T13:25:57Z system:controller:cronjob-controller 2024-03-29T13:25:57Z system:controller:daemon-set-controller 2024-03-29T13:25:57Z system:controller:deployment-controller 2024-03-29T13:25:57Z system:controller:disruption-controller 2024-03-29T13:25:57Z system:controller:endpoint-controller 2024-03-29T13:25:57Z system:controller:endpointslice-controller 2024-03-29T13:25:57Z system:controller:endpointslicemirroring-controller 2024-03-29T13:25:57Z system:controller:ephemeral-volume-controller 2024-03-29T13:25:57Z system:controller:expand-controller 2024-03-29T13:25:57Z system:controller:generic-garbage-collector 2024-03-29T13:25:57Z system:controller:horizontal-pod-autoscaler 2024-03-29T13:25:57Z system:controller:job-controller 2024-03-29T13:25:57Z system:controller:namespace-controller 2024-03-29T13:25:57Z system:controller:node-controller 2024-03-29T13:25:57Z system:controller:persistent-volume-binder 2024-03-29T13:25:57Z system:controller:pod-garbage-collector 2024-03-29T13:25:57Z system:controller:pv-protection-controller 2024-03-29T13:25:57Z system:controller:pvc-protection-controller 2024-03-29T13:25:57Z system:controller:replicaset-controller 2024-03-29T13:25:57Z system:controller:replication-controller 2024-03-29T13:25:57Z system:controller:resourcequota-controller 2024-03-29T13:25:57Z system:controller:root-ca-cert-publisher 2024-03-29T13:25:57Z system:controller:route-controller 2024-03-29T13:25:57Z system:controller:service-account-controller 2024-03-29T13:25:57Z system:controller:service-controller 2024-03-29T13:25:57Z system:controller:statefulset-controller 2024-03-29T13:25:57Z system:controller:ttl-after-finished-controller 2024-03-29T13:25:57Z system:controller:ttl-controller 2024-03-29T13:25:57Z system:coredns 2024-03-29T13:25:59Z system:discovery 2024-03-29T13:25:57Z system:heapster 2024-03-29T13:25:57Z system:kube-aggregator 2024-03-29T13:25:57Z system:kube-controller-manager 2024-03-29T13:25:57Z system:kube-dns 2024-03-29T13:25:57Z system:kube-scheduler 2024-03-29T13:25:57Z system:kubelet-api-admin 2024-03-29T13:25:57Z system:monitoring 2024-03-29T13:25:57Z system:node 2024-03-29T13:25:57Z system:node-bootstrapper 2024-03-29T13:25:57Z system:node-problem-detector 2024-03-29T13:25:57Z system:node-proxier 2024-03-29T13:25:57Z system:persistent-volume-provisioner 2024-03-29T13:25:57Z system:public-info-viewer 2024-03-29T13:25:57Z system:service-account-issuer-discovery 2024-03-29T13:25:57Z system:volume-scheduler 2024-03-29T13:25:57Z tensorboard-controller-manager-role 2024-03-30T00:22:22Z tensorboard-controller-metrics-reader 2024-03-30T00:22:22Z tensorboard-controller-proxy-role 2024-03-30T00:22:22Z tensorboards-web-app-cluster-role 2024-03-30T00:22:22Z tensorboards-web-app-kubeflow-tensorboard-ui-admin 2024-03-30T00:22:22Z tensorboards-web-app-kubeflow-tensorboard-ui-edit 2024-03-30T00:22:22Z tensorboards-web-app-kubeflow-tensorboard-ui-view 2024-03-30T00:22:22Z training-operator 2024-03-30T00:22:22Z view 2024-03-29T13:25:57Z volumes-web-app-cluster-role 2024-03-30T00:22:22Z volumes-web-app-kubeflow-volume-ui-admin 2024-03-30T00:22:22Z volumes-web-app-kubeflow-volume-ui-edit 2024-03-30T00:22:22Z volumes-web-app-kubeflow-volume-ui-view 2024-03-30T00:22:22Z master@k8s-master:$ kubectl get rolebinding -n kubeflow NAME ROLE AGE allow-kubeflow-user-example-com-kubeflow-edit ClusterRole/kubeflow-edit 25d argo-binding Role/argo-role 27d centraldashboard Role/centraldashboard 27d jupyter-web-app-jupyter-notebook-role-binding Role/jupyter-web-app-jupyter-notebook-role 27d kserve-leader-election-rolebinding Role/kserve-leader-election-role 27d kubeflow-pipelines-cache-binding Role/kubeflow-pipelines-cache-role 27d kubeflow-pipelines-metadata-writer-binding Role/kubeflow-pipelines-metadata-writer-role 27d ml-pipeline Role/ml-pipeline 27d ml-pipeline-persistenceagent-binding Role/ml-pipeline-persistenceagent-role 27d ml-pipeline-scheduledworkflow-binding Role/ml-pipeline-scheduledworkflow-role 27d ml-pipeline-ui Role/ml-pipeline-ui 27d ml-pipeline-viewer-crd-binding Role/ml-pipeline-viewer-controller-role 27d notebook-controller-leader-election-rolebinding Role/notebook-controller-leader-election-role 27d pipeline-runner-binding Role/pipeline-runner 27d profiles-leader-election-rolebinding Role/profiles-leader-election-role 27d pvcviewer-leader-election-rolebinding Role/pvcviewer-leader-election-role 27d tensorboard-controller-leader-election-rolebinding Role/tensorboard-controller-leader-election-role 27d master@k8s-master:$ kubectl get clusterrole -n kubeflow-user-example-com NAME CREATED AT addressable-resolver 2024-03-30T00:22:21Z admin 2024-03-29T13:25:57Z admission-webhook-cluster-role 2024-03-30T00:22:21Z admission-webhook-kubeflow-poddefaults-admin 2024-03-30T00:22:21Z admission-webhook-kubeflow-poddefaults-edit 2024-03-30T00:22:21Z admission-webhook-kubeflow-poddefaults-view 2024-03-30T00:22:21Z aggregate-to-kubeflow-pipelines-edit 2024-03-30T00:22:21Z aggregate-to-kubeflow-pipelines-view 2024-03-30T00:22:21Z argo-aggregate-to-admin 2024-03-30T00:22:21Z argo-aggregate-to-edit 2024-03-30T00:22:21Z argo-aggregate-to-view 2024-03-30T00:22:21Z argo-cluster-role 2024-03-30T00:22:21Z authn-delegator 2024-03-29T16:24:26Z broker-addressable-resolver 2024-03-30T00:22:21Z builtin-podspecable-binding 2024-03-30T00:22:21Z calico-kube-controllers 2024-03-29T13:29:37Z calico-node 2024-03-29T13:29:37Z centraldashboard 2024-03-30T00:22:21Z cert-manager-cainjector 2024-03-30T00:22:21Z cert-manager-controller-approve:cert-manager-io 2024-03-30T00:22:21Z cert-manager-controller-certificates 2024-03-30T00:22:21Z cert-manager-controller-certificatesigningrequests 2024-03-30T00:22:21Z cert-manager-controller-challenges 2024-03-30T00:22:21Z cert-manager-controller-clusterissuers 2024-03-30T00:22:21Z cert-manager-controller-ingress-shim 2024-03-30T00:22:21Z cert-manager-controller-issuers 2024-03-30T00:22:21Z cert-manager-controller-orders 2024-03-30T00:22:21Z cert-manager-edit 2024-03-30T00:22:21Z cert-manager-view 2024-03-30T00:22:21Z cert-manager-webhook:subjectaccessreviews 2024-03-30T00:22:21Z channel-addressable-resolver 2024-03-30T00:22:21Z channelable-manipulator 2024-03-30T00:22:21Z cluster-admin 2024-03-29T13:25:57Z dex 2024-03-30T00:22:21Z edit 2024-03-29T13:25:57Z eventing-broker-filter 2024-03-30T00:22:21Z eventing-broker-ingress 2024-03-30T00:22:21Z eventing-config-reader 2024-03-30T00:22:21Z eventing-sources-source-observer 2024-03-30T00:22:21Z flows-addressable-resolver 2024-03-30T00:22:21Z gpu-operator 2024-03-29T14:57:08Z gpu-operator-node-feature-discovery 2024-03-29T14:57:08Z gpu-operator-node-feature-discovery-gc 2024-03-29T14:57:08Z istio-reader-clusterrole-istio-system 2024-03-30T00:22:21Z istio-reader-istio-system 2024-03-30T00:22:21Z istiod-clusterrole-istio-system 2024-03-30T00:22:21Z istiod-gateway-controller-istio-system 2024-03-30T00:22:21Z istiod-istio-system 2024-03-30T00:22:21Z jupyter-web-app-cluster-role 2024-03-30T00:22:21Z jupyter-web-app-kubeflow-notebook-ui-admin 2024-03-30T00:22:21Z jupyter-web-app-kubeflow-notebook-ui-edit 2024-03-30T00:22:21Z jupyter-web-app-kubeflow-notebook-ui-view 2024-03-30T00:22:21Z katib-controller 2024-03-30T00:22:21Z katib-ui 2024-03-30T00:22:21Z knative-bindings-namespaced-admin 2024-03-30T00:22:21Z knative-eventing-controller 2024-03-30T00:22:21Z knative-eventing-namespaced-admin 2024-03-30T00:22:21Z knative-eventing-namespaced-edit 2024-03-30T00:22:21Z knative-eventing-namespaced-view 2024-03-30T00:22:21Z knative-eventing-pingsource-mt-adapter 2024-03-30T00:22:21Z knative-eventing-sources-controller 2024-03-30T00:22:21Z knative-eventing-webhook 2024-03-30T00:22:21Z knative-flows-namespaced-admin 2024-03-30T00:22:21Z knative-messaging-namespaced-admin 2024-03-30T00:22:21Z knative-serving-addressable-resolver 2024-03-30T00:22:21Z knative-serving-admin 2024-03-30T00:22:21Z knative-serving-aggregated-addressable-resolver 2024-03-30T00:22:21Z knative-serving-core 2024-03-30T00:22:21Z knative-serving-istio 2024-03-30T00:22:21Z knative-serving-namespaced-admin 2024-03-30T00:22:21Z knative-serving-namespaced-edit 2024-03-30T00:22:21Z knative-serving-namespaced-view 2024-03-30T00:22:21Z knative-serving-podspecable-binding 2024-03-30T00:22:21Z knative-sources-namespaced-admin 2024-03-30T00:22:21Z kserve-manager-role 2024-03-30T00:22:22Z kserve-models-web-app-cluster-role 2024-03-30T00:22:22Z kserve-proxy-role 2024-03-30T00:22:22Z kubeadm:get-nodes 2024-03-29T13:25:59Z kubeflow-admin 2024-03-30T00:22:22Z kubeflow-edit 2024-03-30T00:22:22Z kubeflow-istio-admin 2024-03-30T00:22:22Z kubeflow-istio-edit 2024-03-30T00:22:22Z kubeflow-istio-view 2024-03-30T00:22:22Z kubeflow-katib-admin 2024-03-30T00:22:22Z kubeflow-katib-edit 2024-03-30T00:22:22Z kubeflow-katib-view 2024-03-30T00:22:22Z kubeflow-kserve-admin 2024-03-30T00:22:22Z kubeflow-kserve-edit 2024-03-30T00:22:22Z kubeflow-kserve-view 2024-03-30T00:22:22Z kubeflow-kubernetes-admin 2024-03-30T00:22:22Z kubeflow-kubernetes-edit 2024-03-30T00:22:22Z kubeflow-kubernetes-view 2024-03-30T00:22:22Z kubeflow-pipelines-cache-role 2024-03-30T00:22:22Z kubeflow-pipelines-edit 2024-03-30T00:22:22Z kubeflow-pipelines-metadata-writer-role 2024-03-30T00:22:22Z kubeflow-pipelines-view 2024-03-30T00:22:22Z kubeflow-training-admin 2024-03-30T00:22:22Z kubeflow-training-edit 2024-03-30T00:22:22Z kubeflow-training-view 2024-03-30T00:22:22Z kubeflow-view 2024-03-30T00:22:22Z kubernetes-dashboard-metrics-scraper 2024-04-04T09:57:55Z local-path-provisioner-role 2024-03-29T14:01:51Z meta-channelable-manipulator 2024-03-30T00:22:22Z ml-pipeline 2024-03-30T00:22:22Z ml-pipeline-persistenceagent-role 2024-03-30T00:22:22Z ml-pipeline-scheduledworkflow-role 2024-03-30T00:22:22Z ml-pipeline-ui 2024-03-30T00:22:22Z ml-pipeline-viewer-controller-role 2024-03-30T00:22:22Z notebook-controller-kubeflow-notebooks-admin 2024-03-30T00:22:22Z notebook-controller-kubeflow-notebooks-edit 2024-03-30T00:22:22Z notebook-controller-kubeflow-notebooks-view 2024-03-30T00:22:22Z notebook-controller-role 2024-03-30T00:22:22Z nvidia-device-plugin 2024-03-29T14:57:33Z nvidia-gpu-feature-discovery 2024-03-29T14:57:33Z nvidia-mig-manager 2024-03-29T14:57:33Z nvidia-operator-validator 2024-03-29T14:57:33Z podspecable-binding 2024-03-30T00:22:22Z pvcviewer-metrics-reader 2024-03-30T00:22:22Z pvcviewer-proxy-role 2024-03-30T00:22:22Z pvcviewer-role 2024-03-30T00:22:22Z seldon-manager-role-seldon-system 2024-04-24T13:38:19Z seldon-manager-sas-role-seldon-system 2024-04-24T13:38:19Z seldon-spartakus-volunteer-seldon-system 2024-04-24T13:38:19Z service-addressable-resolver 2024-03-30T00:22:22Z serving-addressable-resolver 2024-03-30T00:22:22Z source-observer 2024-03-30T00:22:22Z system:aggregate-to-admin 2024-03-29T13:25:57Z system:aggregate-to-edit 2024-03-29T13:25:57Z system:aggregate-to-view 2024-03-29T13:25:57Z system:auth-delegator 2024-03-29T13:25:57Z system:basic-user 2024-03-29T13:25:57Z system:certificates.k8s.io:certificatesigningrequests:nodeclient 2024-03-29T13:25:57Z system:certificates.k8s.io:certificatesigningrequests:selfnodeclient 2024-03-29T13:25:57Z system:certificates.k8s.io:kube-apiserver-client-approver 2024-03-29T13:25:57Z system:certificates.k8s.io:kube-apiserver-client-kubelet-approver 2024-03-29T13:25:57Z system:certificates.k8s.io:kubelet-serving-approver 2024-03-29T13:25:57Z system:certificates.k8s.io:legacy-unknown-approver 2024-03-29T13:25:57Z system:controller:attachdetach-controller 2024-03-29T13:25:57Z system:controller:certificate-controller 2024-03-29T13:25:57Z system:controller:clusterrole-aggregation-controller 2024-03-29T13:25:57Z system:controller:cronjob-controller 2024-03-29T13:25:57Z system:controller:daemon-set-controller 2024-03-29T13:25:57Z system:controller:deployment-controller 2024-03-29T13:25:57Z system:controller:disruption-controller 2024-03-29T13:25:57Z system:controller:endpoint-controller 2024-03-29T13:25:57Z system:controller:endpointslice-controller 2024-03-29T13:25:57Z system:controller:endpointslicemirroring-controller 2024-03-29T13:25:57Z system:controller:ephemeral-volume-controller 2024-03-29T13:25:57Z system:controller:expand-controller 2024-03-29T13:25:57Z system:controller:generic-garbage-collector 2024-03-29T13:25:57Z system:controller:horizontal-pod-autoscaler 2024-03-29T13:25:57Z system:controller:job-controller 2024-03-29T13:25:57Z system:controller:namespace-controller 2024-03-29T13:25:57Z system:controller:node-controller 2024-03-29T13:25:57Z system:controller:persistent-volume-binder 2024-03-29T13:25:57Z system:controller:pod-garbage-collector 2024-03-29T13:25:57Z system:controller:pv-protection-controller 2024-03-29T13:25:57Z system:controller:pvc-protection-controller 2024-03-29T13:25:57Z system:controller:replicaset-controller 2024-03-29T13:25:57Z system:controller:replication-controller 2024-03-29T13:25:57Z system:controller:resourcequota-controller 2024-03-29T13:25:57Z system:controller:root-ca-cert-publisher 2024-03-29T13:25:57Z system:controller:route-controller 2024-03-29T13:25:57Z system:controller:service-account-controller 2024-03-29T13:25:57Z system:controller:service-controller 2024-03-29T13:25:57Z system:controller:statefulset-controller 2024-03-29T13:25:57Z system:controller:ttl-after-finished-controller 2024-03-29T13:25:57Z system:controller:ttl-controller 2024-03-29T13:25:57Z system:coredns 2024-03-29T13:25:59Z system:discovery 2024-03-29T13:25:57Z system:heapster 2024-03-29T13:25:57Z system:kube-aggregator 2024-03-29T13:25:57Z system:kube-controller-manager 2024-03-29T13:25:57Z system:kube-dns 2024-03-29T13:25:57Z system:kube-scheduler 2024-03-29T13:25:57Z system:kubelet-api-admin 2024-03-29T13:25:57Z system:monitoring 2024-03-29T13:25:57Z system:node 2024-03-29T13:25:57Z system:node-bootstrapper 2024-03-29T13:25:57Z system:node-problem-detector 2024-03-29T13:25:57Z system:node-proxier 2024-03-29T13:25:57Z system:persistent-volume-provisioner 2024-03-29T13:25:57Z system:public-info-viewer 2024-03-29T13:25:57Z system:service-account-issuer-discovery 2024-03-29T13:25:57Z system:volume-scheduler 2024-03-29T13:25:57Z tensorboard-controller-manager-role 2024-03-30T00:22:22Z tensorboard-controller-metrics-reader 2024-03-30T00:22:22Z tensorboard-controller-proxy-role 2024-03-30T00:22:22Z tensorboards-web-app-cluster-role 2024-03-30T00:22:22Z tensorboards-web-app-kubeflow-tensorboard-ui-admin 2024-03-30T00:22:22Z tensorboards-web-app-kubeflow-tensorboard-ui-edit 2024-03-30T00:22:22Z tensorboards-web-app-kubeflow-tensorboard-ui-view 2024-03-30T00:22:22Z training-operator 2024-03-30T00:22:22Z view 2024-03-29T13:25:57Z volumes-web-app-cluster-role 2024-03-30T00:22:22Z volumes-web-app-kubeflow-volume-ui-admin 2024-03-30T00:22:22Z volumes-web-app-kubeflow-volume-ui-edit 2024-03-30T00:22:22Z volumes-web-app-kubeflow-volume-ui-view 2024-03-30T00:22:22Z master@k8s-master:$ kubectl get rolebinding -n kubeflow-user-example-com NAME ROLE AGE default-editor ClusterRole/kubeflow-edit 27d default-viewer ClusterRole/kubeflow-view 27d namespaceAdmin ClusterRole/kubeflow-admin 27d sa-pipeline-runner Role/pipeline-runner 27d user-pipeline-runner Role/pipeline-runner 27d

[rimolive]

What is the Kubeflow version you have installed?

[tugasakhirai21]

@rimolive i installed latest kubeflow release v1.8.1

[ahxxm]

We are affected too, v1.7 on AWS

In addition to removing the namespace param, replacing the value with current authenticated username will also enable downloading artifact, e.g. ?namespace=ahxxm

Inspired by this comment, I applied the following patch and it fixed the problem, I can head and download logs from UI

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-all-pipeline-artifacts
  namespace: test-sample
spec:
  action: ALLOW
  rules:
    - to:
        - operation:
            paths: ["/pipeline/artifacts/*"]

the test-sample namespace should be replaced by the one that triggers RBAC error

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ahxxm]

please don't close

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ahxxm]

please don't close

[rimolive]

@juliusvonkohout FYI. I believe we can apply @ahxxm fix onto Istio AuthorizationPolicy to avoid RBAC denied issues.

[juliusvonkohout]

Is it still applicable to Kubeflow 1.9.x?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[juliusvonkohout]

Hello, to get support please verify this on Kubeflow 1.9.1 https://github.com/kubeflow/manifests?tab=readme-ov-file#installation. 1.7-1.8.1 is too old. There were many authentication changes in 1.9.1 and so far it is working for me.