kubeflow 1.3 not working with rbac on microk8s or minikube

[lukemarsden]

kubeflow 1.3 + microk8s 1.21 + rbac

microk8s enable rbac

then

microk8s enable kubeflow

leads to a hang on

Waited 9510s for operator pods to come up, 1 remaining.

that's

istio-ingressgateway                                       waiting      1  istio-ingressgateway  jujucharms   20  kubernetes                  Waiting for Istio Pilot information

also:

luke@mind:~$ microk8s kubectl auth can-i get configmaps --namespace kf --as kubeflow-dashboard
no

kubeflow 1.3 + minikube installing k8s 1.21

upon logging in (after working around https://github.com/canonical/bundle-kubeflow/issues/351#issuecomment-830044248 by carefully updating the config of oidc-gatekeeper second)

kf                      kubeflow-dashboard-f848cb4bb-2zf2k           0/1     Init:0/1      0          41s

errors with:

Unable to fetch ConfigMap: {
  kind: 'Status',
  apiVersion: 'v1',
  metadata: {},
  status: 'Failure',
  message: 'configmaps "centraldashboard-config" not found',
  reason: 'NotFound',
  details: { name: 'centraldashboard-config', kind: 'configmaps' },
  code: 404
}

and the UI errors with configmaps "centraldashboard-config" not found

root@25d86afde7f0a21a:~# kubectl auth can-i get configmaps --namespace kf --as kubeflow-dashboard
no

root@25d86afde7f0a21a:~# kubectl get clusterrolebindings -A |grep dashboard
kubeflow-dashboard-kf-kubeflow-dashboard                 2m55s

the reason for the inability of the dashboard to actually see the configmap is unclear (wrong namespace? wrong serviceaccount?), since:

root@25d86afde7f0a21a:~# kubectl get clusterrolebinding kubeflow-dashboard-kf-kubeflow-dashboard -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    controller.juju.is/id: 040c9fce-b297-422d-8c5c-a63bd1611253
    model.juju.is/id: a9de03b1-7b21-4fb6-855d-7b33f2bfc125
  creationTimestamp: "2021-05-19T07:56:31Z"
  labels:
    app.juju.is/created-by: controller
    app.kubernetes.io/managed-by: juju
    app.kubernetes.io/name: kubeflow-dashboard
    model.juju.is/name: kf
  name: kubeflow-dashboard-kf-kubeflow-dashboard
  resourceVersion: "290153"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/kubeflow-dashboard-kf-kubeflow-dashboard
  uid: b7ef33e4-74e5-4778-a9a2-ee5e9375016c
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kf-kubeflow-dashboard
subjects:
- kind: ServiceAccount
  name: kubeflow-dashboard
  namespace: kf
root@25d86afde7f0a21a:~# kubectl get clusterrole kf-kubeflow-dashboard -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    controller.juju.is/id: 040c9fce-b297-422d-8c5c-a63bd1611253
    model.juju.is/id: a9de03b1-7b21-4fb6-855d-7b33f2bfc125
  creationTimestamp: "2021-05-18T16:02:24Z"
  labels:
    app.juju.is/created-by: controller
    app.kubernetes.io/managed-by: juju
    app.kubernetes.io/name: kubeflow-dashboard
    model.juju.is/name: kf
  name: kf-kubeflow-dashboard
  resourceVersion: "2681"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/kf-kubeflow-dashboard
  uid: 41c9da44-5cb9-493b-ab72-29d112962ded
rules:
- apiGroups:
  - ""
  resources:
  - events
  - namespaces
  - nodes
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  - app.k8s.io
  resources:
  - applications
  - pods
  - pods/exec
  - pods/log
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - secrets
  - configmaps
  verbs:
  - get

BTW: the fact that microk8s ships with rbac disabled by default doesn't help here, because it means that rbac-related issues (rbac is enabled in almost all k8s clusters) get hidden from common usage with microk8s.

[DomFleischmann]

Regarding the istio-ingressgateway charm being stuck in a waiting status this additional step is needed for it to work in an rbac environment: kubectl patch role -n kf istio-ingressgateway-operator -p '{"apiVersion":"rbac.authorization.k8s.io/v1","kind":"Role","metadata":{"name":"istio-ingressgateway-operator"},"rules":[{"apiGroups":["*"],"resources":["*"],"verbs":["*"]}]}'

This was accidentally removed from our documentation and will be re-added.

The kubeflow-dashboard config map requires some further investigation for a fix.

[evilnick]

The docs have been updated now

[DomFleischmann]

I haven't been able to reproduce the kubeflow-dashboard issue with the latest kubeflow bundle on microk8s 1.20/candidate. @lukemarsden Can you test this again with the latest bundle on your environment and see if you can reproduce it still?