influxdata / tick-charts

A repository for Helm Charts for the full TICK Stack
Apache License 2.0
90 stars 74 forks source link

/stats/summary returned HTTP status 401 Unauthorized #77

Open casertap opened 5 years ago

casertap commented 5 years ago

Using this config in telegraf-ds

[[inputs.kubernetes]]
  bearer_token = "/var/run/secrets/kubernetes.io/serviceaccount/token"
  insecure_skip_verify = true
  url = "https://$HOSTIP:10250"

I get this error:

Jan 18 15:40:10 hosts-telegraf-ds-wsnbp hosts-telegraf-ds Error E! [inputs.kubernetes]: Error in plugin: https://172.23.47.179:10250/stats/summary returned HTTP status 401 Unauthorized

It seems that I am missing the serviceaccount token for telegraf to query the kubernetes endpoint.

I find it weird that the tick-charts did not create this serviceaccount automatically.

What am I missing? How can I make the inputs.kubernetes work?

jackzampolin commented 5 years ago

@casertap when I wrote these charts serviceaccounts weren't a thing yet. You may need to add one to this chart.

rawkode commented 5 years ago

@casertap I don't suppose you're running this on GKE?

casertap commented 5 years ago

@rawkode no I built my own kube cluster on aws using kops

rawkode commented 5 years ago

@casertap Please ensure you have Webhook authentication enabled in your Kubelet configuration:

--authentication-token-webhook

niklasember commented 5 years ago

@rawkode I'm having this issue while running on GKE, any ideas?

Tried with --authentication-token-webhook on kubelet and have created a serviceaccount. Same config works on non-gke setup.

rawkode commented 5 years ago

@niklasember GKE doesn't allow access to Kubelet on the host, you need to go through API Server

- kubernetes:
        url: "http://kubernetes.default.svc.cluster.local/v1/nodes/${HOSTIP}/proxy/metrics"
        bearer_token: "/var/run/secrets/kubernetes.io/serviceaccount/token"
florianrusch commented 5 years ago

@jackzampolin do you have an example how to configure the role for the service account?

florianrusch commented 5 years ago

I've found a solution that works for me:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tick-stack

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: metric-scanner-kubelet-api-admin
subjects:
- kind: ServiceAccount
  name: tick-stack
  namespace: tick
roleRef:
  kind: ClusterRole
  name: system:kubelet-api-admin
  apiGroup: rbac.authorization.k8s.io
florianrusch commented 5 years ago

We should update the telegraf-ds chart to also create this service-account automatically.

rawkode commented 5 years ago

@florianrusch I agree. We're working on cleaning up our Helm charts and will be supporting / endorsing the charts in the official Helm repository very soon

pbaderia01 commented 5 years ago

@florianrusch Would it be possible for you to list out the steps that you followed to get the service account working for you?

florianrusch commented 5 years ago

@piyush-insider I didn't test it again. But I've took the resources I've published before and just applied them to the cluster/namespace.

You can put the resources in a yaml-file and kubectl apply this file, or you can run this command:

$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tick-stack

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: metric-scanner-kubelet-api-admin
subjects:
- kind: ServiceAccount
  name: tick-stack
  namespace: tick
roleRef:
  kind: ClusterRole
  name: system:kubelet-api-admin
  apiGroup: rbac.authorization.k8s.io
EOF
stanislav-zaprudskiy commented 5 years ago

Apart from service account specification (which also requires #105) you'll need to provide adequate ClusterRole and ClusterRoleBinding.

For kubernetes plugin to work, assigning service account to system:kubelet-api-admin role (referenced above) is too much. In my case what was enough is something like below:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: telegraf
  namespace: monitoring
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: metrics-reader
rules:
- apiGroups: [""]
  resources: ["nodes/stats"]
  verbs: ["get"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: telegraf-metrics-reader
subjects:
- kind: ServiceAccount
  name: telegraf
  namespace: monitoring
roleRef:
  kind: ClusterRole
  name: metrics-reader
  apiGroup: rbac.authorization.k8s.io