yxxhero
commented
6 years ago --requestheader-allowed-names=system:kube-controller-manager
Open 4220182 opened 6 years ago
yxxhero
commented
6 years ago --requestheader-allowed-names=system:kube-controller-manager
Xfireman
commented
6 years ago 我在日志里也有同样的报错，但是我获取资源都是正常的。
miaoxiaoy
commented
6 years ago 从源码里复制出来的resource-reader.yaml这个文件中设置metric-server可访问apiserver的资源的授权有写问题造成的.
注意下面的注释说明
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: system:metrics-server
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups:
- metrics.k8s.io ########这里将原来的""换成 "metrics.k8s.io"
resources:
- pods
- nodes
- namespaces
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
resources:
- deployments
verbs:
- get
- list
- update
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: system:metrics-server
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:metrics-server
subjects:
#- kind: ServiceAccount
- kind: User ############## 把ServiceAccount改为User
name: metrics-server ######### 创建metric-server证书的时候, 将CN写成这里的"metrics-server"
namespace: kube-system注意 apiserver的启动参数--requestheader-allowed-names 留空"" 代表允许任何用户访问apiserver(授权是另外一回事, 这个和RBAC没啥关系) 在这里可以留空或者写成ClusterRoleBinding里subject里的用户名
最后 kubectl apply -f resource-reader.yaml
测试方法 kubectl top pods
[k8s@kube-node1 metrics-server]$ kubectl top pods
NAME CPU(cores) MEMORY(bytes)
my-nginx-86555897f9-57v4t 0m 2Mi
my-nginx-86555897f9-8mzxq 0m 2Mi
my-nginx-86555897f9-dkm8m 0m 2Mi
nginx-ds-l8qkg 0m 2Mi
nginx-ds-sll9g 0m 2Mi
php-59447fb5c-28dwl 0m 10Mi
php-59447fb5c-g6pwr 0m 9Mi
php-59447fb5c-phtrh 0m 9Mi --requestheader-allowed-names=aggregator 只要不设置该参数即可，也不会再出现相关报错。不是设置空，是删掉该参数
官方文档中对于该参数的解释 --requestheader-allowed-names stringSlice | List of client certificate common names to allow to provide usernames in headers specified by --requestheader-username-headers. If empty, any client certificate validated by the authorities in --requestheader-client-ca-file is allowed.
4220182
commented
6 years ago @miaoxiaoy 我按照你的方法，还是有问题：
$ kubectl logs metrics-server-v0.2.1-66b95ddf44-2th8c -n kube-system -c metrics-server
E0725 06:20:30.577090 1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/processors/namespace_based_enricher.go:85: Failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list namespaces at the cluster scope
E0725 06:20:30.615137 1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/heapster.go:254: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list pods at the cluster scope
E0725 06:20:31.581478 1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/processors/namespace_based_enricher.go:85: Failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list namespaces at the cluster scope
E0725 06:20:31.617166 1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/heapster.go:254: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list pods at the cluster scope
E0725 06:20:32.597102 1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/processors/namespace_based_enricher.go:85: Failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list namespaces at the cluster scope
E0725 06:20:32.619230 1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/heapster.go:254: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list pods at the cluster scope@4220182 heapster源码里的授权文件权限有点问题
cat /home/k8s/k8s-install/heapster-1.5.3/deploy/kube-config/rbac/heapster-rbac.yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: heapster-kubelet-api
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:kubelet-api-admin
subjects:
- kind: ServiceAccount
name: heapster
namespace: kube-system原因是system:kubelet-api-admin这个clusterrole角色的权限不足, 命令行直接修改system:kubelet-api-admin这个集群角色
[k8s@kube-node1 hpa]$ kubectl edit clusterrole system:kubelet-api-admin
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
creationTimestamp: 2018-07-04T11:17:42Z
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:kubelet-api-admin
resourceVersion: "1989133"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/system%3Akubelet-api-admin
uid: debcb355-7f7b-11e8-b7a6-005056860efb
rules:
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- proxy
- apiGroups:
- ""
resources:
- nodes/log
- nodes/metrics
- nodes/proxy
- nodes/spec
- nodes/stats
verbs:
- '*'
#################################### 下面是新增的, 增加了对pod, namespace的 get, list, watch 权限
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- namespaces
verbs:
- get
- list
- watch再请教一下，metrics-server 需要先安装heapster 插件吗？ @miaoxiaoy
miaoxiaoy
commented
6 years ago @4220182 是的
KielChan
commented
5 years ago 从源码里复制出来的resource-reader.yaml这个文件中设置metric-server可访问apiserver的资源的授权有写问题造成的.
解决方法, 修改resource-reader.yaml:
注意下面的注释说明
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: system:metrics-server labels: kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile rules: - apiGroups: - metrics.k8s.io ########这里将原来的""换成 "metrics.k8s.io" resources: - pods - nodes - namespaces verbs: - get - list - watch - apiGroups: - "extensions" resources: - deployments verbs: - get - list - update - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: system:metrics-server labels: kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:metrics-server subjects: #- kind: ServiceAccount - kind: User ############## 把ServiceAccount改为User name: metrics-server ######### 创建metric-server证书的时候, 将CN写成这里的"metrics-server" namespace: kube-system注意 apiserver的启动参数--requestheader-allowed-names 留空"" 代表允许任何用户访问apiserver(授权是另外一回事, 这个和RBAC没啥关系) 在这里可以留空或者写成ClusterRoleBinding里subject里的用户名
最后 kubectl apply -f resource-reader.yaml
测试方法 kubectl top pods
[k8s@kube-node1 metrics-server]$ kubectl top pods NAME CPU(cores) MEMORY(bytes) my-nginx-86555897f9-57v4t 0m 2Mi my-nginx-86555897f9-8mzxq 0m 2Mi my-nginx-86555897f9-dkm8m 0m 2Mi nginx-ds-l8qkg 0m 2Mi nginx-ds-sll9g 0m 2Mi php-59447fb5c-28dwl 0m 10Mi php-59447fb5c-g6pwr 0m 9Mi php-59447fb5c-phtrh 0m 9Mi
My metrics-server setup without heapster, and it can work fine. But, after following your steps, metrics-server cannot work, and the warnings in kube-apiserver still exist.
zuihou
commented
5 years ago kubectl edit clusterrole system:kubelet-api-admin
按照您说的 执行 $ kubectl edit clusterrole system:kubelet-api-admin 修改权限后，确实可以了. /apis/metrics.k8s.io/v1beta1/nodes /apis/metrics.k8s.io/v1beta1/pods 都有数据了。
但这种手动修改的权限， 重启后会不会重置呀? 有没有永久修改的方法呀
按照： 09-4.metrics-server插件这篇说明 ，在增加kube-apiserver.service 增加以下：
kube-controllr-manager.service 增加：
重启apiserver和controller-manager之后，在kube-apiserver的日志一直有错误（x509)：
证书也是按照：创建 metrics-server 证书签名请求 做的证书。
metrcs-server 输出的 metrics ：
不知道是问题出在哪里？为什么直接使用 kubectl 命令访问，会报错？kube-apiserver里面的x509警告是哪里导致的？