How to use Bearer token to connect karmada proxy server

631068264 commented 2 years ago

What happened:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "serviceaccounts \"custom-admin\" is forbidden: User \"system:serviceaccount:karmada-cluster:karmada-impersonator\" cannot impersonate resource \"serviceaccounts\" in API group \"\" in the namespace \"karmada-custom-user\"",
  "reason": "Forbidden",
  "details": {
    "name": "custom-admin",
    "kind": "serviceaccounts"
  },
  "code": 403
}

How to reproduce it (as minimally and precisely as possible): Follow this https://karmada.io/docs/userguide/globalview/aggregated-api-endpoint/ to write custom rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: custom-admin
  namespace: karmada-custom-user

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: custom-cluster-proxy-clusterrole
rules:
- apiGroups:
  - 'cluster.karmada.io'
  resources:
  - clusters/proxy
  resourceNames:
  - member1
  - member2
  verbs:
  - '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: custom-cluster-proxy-clusterrolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: custom-cluster-proxy-clusterrole
subjects:
- kind: Group
  name: system:serviceaccounts:karmada-custom-user

Anything else we need to know?:

Environment: Karmada version: 1.2.1 kubectl-karmada or karmadactl version (the result of kubectl-karmada version or karmadactl version): karmadactl version: version.Info{GitVersion:"v1.2.1", GitCommit:"de4972b74f848f78a58f9a0f4a4e85f243ba48f8", GitTreeState:"clean", BuildDate:"2022-07-14T09:33:33Z", GoVersion:"go1.17.11", Compiler:"gc", Platform:"linux/amd64"} Others:

lonelyCZ commented 2 years ago

Please try

subjects:
- kind: ServiceAccount
  name: custom-admin
  namespace: karmada-custom-user

XiShanYongYe-Chang commented 2 years ago

Make a try with the following ClusterRoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: custom-cluster-proxy-clusterrolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: custom-cluster-proxy-clusterrole
subjects:
- kind: Group
  name: system:serviceaccounts:karmada-custom-user
- kind: User
  name: system:serviceaccounts

631068264 commented 2 years ago

Please try

subjects:
- kind: ServiceAccount
  name: custom-admin
  namespace: karmada-custom-user

@lonelyCZ I have tried it get similar error too，I forgot to say

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "groups \"system:serviceaccounts\" is forbidden: User \"system:serviceaccount:karmada-cluster:karmada-impersonator\" cannot impersonate resource \"groups\" in API group \"\" at the cluster scope",
  "reason": "Forbidden",
  "details": {
    "name": "system:serviceaccounts",
    "kind": "groups"
  },
  "code": 403
}[

631068264 commented 2 years ago

subjects:
- kind: Group
  name: system:serviceaccounts:karmada-custom-user
- kind: User
  name: system:serviceaccounts

@XiShanYongYe-Chang not work

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "serviceaccounts \"custom-admin\" is forbidden: User \"system:serviceaccount:karmada-cluster:karmada-impersonator\" cannot impersonate resource \"serviceaccounts\" in API group \"\" in the namespace \"karmada-custom-user\"",
  "reason": "Forbidden",
  "details": {
    "name": "custom-admin",
    "kind": "serviceaccounts"
  },
  "code": 403
}

XiShanYongYe-Chang commented 2 years ago

Sorry, I write it wrong.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: custom-cluster-proxy-clusterrolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: custom-cluster-proxy-clusterrole
subjects:
- kind: ServiceAccount
  name: custom-admin
  namespace: karmada-custom-user
- kind: Group
  name: system:serviceaccounts
- kind: Group
  name: system:serviceaccounts:karmada-custom-user

XiShanYongYe-Chang commented 2 years ago

You can have a look with https://mp.weixin.qq.com/s?__biz=MzIzNzU5NTYzMA==&mid=2247497323&idx=1&sn=4dd7eb1966c7923a3317d036a385fd52#rd

631068264 commented 2 years ago

好像有点用，但又不完全有 @XiShanYongYe-Chang

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "nodes is forbidden: User \"system:serviceaccount:karmada-custom-user:custom-admin\" cannot list resource \"nodes\" in API group \"\" at the cluster scope",
  "reason": "Forbidden",
  "details": {
    "kind": "nodes"
  },
  "code": 403
}

curl

curl -k -X GET 'https://xxxxxx:5443/apis/cluster.karmada.io/v1alpha1/clusters/member1/proxy/api/v1/nodes' \
-H 'Authorization: Bearer  xxxxx'

XiShanYongYe-Chang commented 2 years ago

How about change:

- kind: ServiceAccount
  name: custom-admin
  namespace: karmada-custom-user

to

- kind: User
  name: system:serviceaccount:karmada-custom-user:custom-admin

631068264 commented 2 years ago

just change

subjects:
- kind: Group
  name: system:serviceaccounts:karmada-custom-user
- kind: Group
  name: system:serviceaccounts
- kind: User
  name: system:serviceaccount:karmada-custom-user:custom-admin

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "serviceaccounts \"custom-admin\" is forbidden: User \"system:serviceaccount:karmada-cluster:karmada-impersonator\" cannot impersonate resource \"serviceaccounts\" in API group \"\" in the namespace \"karmada-custom-user\"",
  "reason": "Forbidden",
  "details": {
    "name": "custom-admin",
    "kind": "serviceaccounts"
  },
  "code": 403
}

XiShanYongYe-Chang commented 2 years ago

Hi @631068264, thank you for trying so many times. I'll probably be busy with something else now, let me come back in a couple of days and try to fix it.

XiShanYongYe-Chang commented 2 years ago

/assign

lonelyCZ commented 2 years ago

Hi @631068264 , seemingly, you need to grant relative permissions to ServiceAccount in the member cluster.

Like

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  creationTimestamp: "2022-08-30T06:34:36Z"
  name: zhangsan
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: zhangsan
  namespace: zhangsan
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: system:serviceaccount:default:zhangsan
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:serviceaccounts:default

It seemingly only proxy the existing permission.

Refering https://mp.weixin.qq.com/s?__biz=MzIzNzU5NTYzMA==&mid=2247497323&idx=1&sn=4dd7eb1966c7923a3317d036a385fd52#rd

假设管理员已经在成员集群中使用ServiceAccount创建了一个名为zhangsan的帐户，我们知道Kubernetes会为该ServiceAccount自动生成token。当管理员为zhangsan通过RBAC配置完权限后，zhangsan就可以访问成员集群，此时zhangsan也仅能访问该成员集群。

631068264 commented 2 years ago

@lonelyCZ 但是还是一样，没什么用。而且这篇文章是Karmada v1.0.0

之前确实是漏设置member 的sa

apiVersion: v1
kind: ServiceAccount
metadata:
  name: custom-admin
  namespace: karmada-custom-user

kubectl --kubeconfig=member1-config get sa -n karmada-custom-user
NAME           SECRETS   AGE
custom-admin   1         13m
default        1         23h

host

apiVersion: v1
kind: ServiceAccount
metadata:
  name: custom-admin
  namespace: karmada-custom-user

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: custom-cluster-proxy-clusterrole
rules:
- apiGroups:
  - 'cluster.karmada.io'
  resources:
  - clusters/proxy
  resourceNames:
  - member1
  - member2
  verbs:
  - '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: custom-cluster-proxy-clusterrolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: custom-cluster-proxy-clusterrole
subjects:
- kind: ServiceAccount
  name: custom-admin
  namespace: karmada-custom-user
- kind: Group
  name: "system:serviceaccounts"
- kind: Group
  name: "system:serviceaccounts:karmada-custom-user"

same error

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "nodes is forbidden: User \"system:serviceaccount:karmada-custom-user:custom-admin\" cannot list resource \"nodes\" in API group \"\" at the cluster scope",
  "reason": "Forbidden",
  "details": {
    "kind": "nodes"
  },
  "code": 403
}

lonelyCZ commented 2 years ago

apiVersion: v1
kind: ServiceAccount
metadata:
  name: custom-admin
  namespace: karmada-custom-user

需要给子集群中的sa配置访问资源的权限，你把下面这个配到子集群中试试。我早上试了一下，是可以的

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  name: custom-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: custom-admin
  namespace: karmada-custom-user
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: system:serviceaccount:karmada-custom-user:custom-admin
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:serviceaccounts:karmada-custom-user

631068264 commented 2 years ago

OK了贴一下完整步骤

member-proxy-rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: custom-admin
  namespace: karmada-custom-user

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  name: custom-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:serviceaccounts:karmada-custom-user

kubectl --kubeconfig=member1-config  -f  member-proxy-rbac.yaml

然后host

apiVersion: v1
kind: ServiceAccount
metadata:
  name: custom-admin
  namespace: karmada-custom-user

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: custom-cluster-proxy-clusterrole
rules:
- apiGroups:
  - 'cluster.karmada.io'
  resources:
  - clusters/proxy
  resourceNames:
  - member1
  - member2
  verbs:
  - '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: custom-cluster-proxy-clusterrolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: custom-cluster-proxy-clusterrole
subjects:
- kind: ServiceAccount
  name: custom-admin
  namespace: karmada-custom-user
- kind: Group
  name: "system:serviceaccounts"
- kind: Group
  name: "system:serviceaccounts:karmada-custom-user"

kubectl --kubeconfig karmada-apiserver.config apply -f cluster-proxy-rbac.yaml

# 获取token
kubectl --kubeconfig karmada-apiserver.config -n karmada-custom-user describe secret $(kubectl -n karmada-custom-user get secret | grep admin | awk '{print $1}')

karmada-io / karmada

How to use Bearer token to connect karmada proxy server #2448