shikanon / kubeflow-manifests

kubeflow国内一键安装文件
GNU General Public License v3.0
338 stars 117 forks source link

notebooks访问pipeline sdk问题(鉴权问题) #36

Closed Culiner closed 3 years ago

Culiner commented 3 years ago

哈喽,这个仓库解决了很多国内安装的过程,非常感谢。 我这边启动完成之后,各个组件都处于运行状态,在kubeflow-user-example-com命名空间下创建了notebook,使用kfp.Client访问pipeline sdk出现各种权限问题,想问下如何能通过kfp.Client访问sdk呢?

shikanon commented 3 years ago

@Culiner 具体是哪步操作出现了权限问题?可以贴示例代码吗

Culiner commented 3 years ago

image 尝试使用kubeflow空间下的service/ml-pipeline端口,但会报错误: image

如果使用istio-system下的service/istio-ingressgateway,获取不到任何信息,并且也无法创建实验 image

Culiner commented 3 years ago

@shikanon 如果不指定host的话,同样出现HTTP response body: RBAC: access denied的问题 不知道如何才能正确的调用,如果能给一个示例就太好了,谢谢~

shikanon commented 3 years ago

@Culiner 可以尝试给你的 notebook pod 加上ml-pipeline的用户权限:

  serviceAccount: ml-pipeline
  serviceAccountName: ml-pipeline

role:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ml-pipeline
  namespace: xxxxx
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - pods/log
  verbs:
  - get
  - list
  - delete
- apiGroups:
  - argoproj.io
  resources:
  - workflows
  verbs:
  - create
  - get
  - list
  - watch
  - update
  - patch
  - delete
- apiGroups:
  - kubeflow.org
  resources:
  - scheduledworkflows
  verbs:
  - create
  - get
  - list
  - update
  - patch
  - delete
- apiGroups:
  - authorization.k8s.io
  resources:
  - subjectaccessreviews
  verbs:
  - create
- apiGroups:
  - authentication.k8s.io
  resources:
  - tokenreviews
  verbs:
  - create
Culiner commented 3 years ago

@shikanon 我试图调整notebook的account,显示Unprocessable Entity,观察到它默认的用户是default-editor,所以我对default-editor加了role,在用户空间kubeflow及kubeflow-user-example-com空间下都加上了,但还是显示同样的问题。不知道是否是因为版本问题? 我的notebook版本是public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-tensorflow-full:v1.3.0-rc.0 我的pipeline server版本是registry.cn-shenzhen.aliyuncs.com/tensorbytes/ml-pipeline-api-server:1.5.0-rc.2-081bf

shikanon commented 3 years ago

@Culiner RBAC不仅仅是命名空间,还有资源的操作权限,如果不是生产环境可以简单粗暴把你的用户和 admin 这个clusterrole做绑定,同时赋予他所有资源的操作权限,像我上面介绍那样,这样可以保证你的账号权限是足够的。

Culiner commented 3 years ago

@shikanon 我注意到,确实是因为所在用户的权限问题导致的,可以正常获取到pipeline列表却无法创建pipeline,并且执行get_user_namespace获取到的是空值,我是个新手,能否帮忙说明一下如何才能给对应的用户和admin这个clusterrole做绑定呢? image

Culiner commented 3 years ago

这边显示的权限报错是

Failed to authorize the request: Failed to authorize with API resource references: PermissionDenied: User 'user@example.com' is not authorized with reason: (request: &ResourceAttributes{Namespace:kubeflow-user-example-com,Verb:create,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:experiments,Subresource:,Name:Default,}): Unauthorized access

shikanon commented 3 years ago

加上这个rolebinding试试:

$ kubectl get rolebinding -n kubeflow-user-example-com namespaceAdmin -oyaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  annotations:
    role: admin
    user: user@example.com
  name: superAdmin
  namespace: kubeflow-user-example-com
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubeflow-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: user@example.com
Culiner commented 3 years ago

@shikanon 加完之后需要重启么?我重启了notebook的pod,但仍然不行。 当我调用list_experiments的时候,显示以下错误 Internal error: Unauthenticated: Request header error: there is no user identity header.: Request header error: there is no user identity header.\nFailed to authorize with API resource references 当我加了以下EnvoyFilter.yaml的时候,就会出现HTTP response body: RBAC: access denied

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: add-header
  namespace: kubeflow-user-example-com
spec:
  configPatches:
  - applyTo: VIRTUAL_HOST
    match:
      context: SIDECAR_OUTBOUND
      routeConfiguration:
        vhost:
          name: ml-pipeline.kubeflow.svc.cluster.local:8888
          route:
            name: default
    patch:
      operation: MERGE
      value:
        request_headers_to_add:
        - append: true
          header:
            key: kubeflow-userid
            value: admin@example.com
  workloadSelector:
    labels:
      notebook-name: jupyter-tensorflow-test

请问调用时是否需要指定namespace呢?我在创建客户端的时候指定了namespace,像client = kfp.Client(namespace='kubeflow'),但调用client.get_user_namespace()仍然得到了空值

shikanon commented 3 years ago

@Culiner 你的用户名到底是 user@example.com 还是 admin@example.com ? 我看你前面报错是 user@example.com 没有权限,这里怎么使用的 admin@exmaple.com

Culiner commented 3 years ago

@shikanon 平台登录时用的是 admin@example.com ,创建kubeflow-user-example-com命名空间的时候owner是 user@example.com,使用的是您仓库里的kubeflow-manifests/manifest1.3/033-user-namespace-user-namespace-base.yaml 两个用户我都尝试了,都会被权限拒绝

shikanon commented 3 years ago

@Culiner 最好统一一下吧,这样再定位错误的时候也可以更快速定位,admin@example.com 是我用来替换掉 033-user-namespace-user-namespace-base.yaml 里面得用户名称的,因为改了他密码,所以干脆将用户也改了,kubeflow 原本用得 user@example.com。

Culiner commented 3 years ago

@shikanon 感谢您的解答,我调整了空间的所属人为admin,然后添加了以下内容就ok了

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  labels:
    app.kubernetes.io/component: ml-pipeline
    app.kubernetes.io/name: kubeflow-pipelines
    application-crd-id: kubeflow-pipelines
  name: ml-pipeline-api
  namespace: kubeflow
spec:
  rules:
  - from:
    - source:
        principals:
        - cluster.local/ns/kubeflow-user-example-com/sa/default-editor
  - when:
    - key: request.headers[kubeflow-userid]
      notValues:
      - '*'
  selector:
    matchLabels:
      app: ml-pipeline
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: add-header
  namespace: kubeflow-user-example-com
spec:
  configPatches:
  - applyTo: VIRTUAL_HOST
    match:
      context: SIDECAR_OUTBOUND
      routeConfiguration:
        vhost:
          name: ml-pipeline.kubeflow.svc.cluster.local:8888
          route:
            name: default
    patch:
      operation: MERGE
      value:
        request_headers_to_add:
        - append: true
          header:
            key: kubeflow-userid
            value: admin@example.com
  workloadSelector:
    labels:
      notebook-name: jupyter-tensorflow-test