prometheus-to-sd --dynamic-source is resolving pods in kube-system

[l15k4]

Hi, I have this DaemonSet :

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: {{ template "prometheus.fullname" . }}
  labels:
    app: {{ template "prometheus.name" . }}
    chart: {{ template "qpipeline.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  selector:
    matchLabels:
      monitor: kamon-to-prometheus
  template:
    metadata:
      labels:
        monitor: kamon-to-prometheus
    spec:
      containers:
        - name: prometheus-to-sd
          image: {{ .Values.stackdriver.image }}
          command: ["/monitor", "--stackdriver-prefix={{ .Values.stackdriver.prefix }}",
                    "--dynamic-source=mix:http://:{{ .Values.stackdriver.port }}{{ .Values.stackdriver.endpoint }}?podIdLabel=kamon-to-prometheus&namespaceIdLabel=default",
                    "--namespace-id=default"]

To run in default namespace and it should resolve pods also in default namespace, but I cannot force it and I get :

main.go:123] pods is forbidden: User "system:serviceaccount:default:default" cannot list pods in the namespace "kube-system": Unknown user "system:serviceaccount:default:default"

which means it tries to do service discovery in the kube-system instead of default ... It is hardcoded here for kube-system https://github.com/GoogleCloudPlatform/k8s-stackdriver/blob/edcfb0bfdb9b25422ed235615b39a4c024d9de09/prometheus-to-sd/config/dynamic_source.go#L17

I mean, I'm running this DaemonSet in default namespace and all pods that it should discover lives also in default namespace, but it has hardcoded kube-system namespace. Shouldn't it use the --namespace-id flag instead of it being a constant?

[edevil]

Did you manage to get around this issue?

[l15k4]

@edevil No, I didn't, I use --source

[bmenasha]

additonally prometheus-to-sd only seems to perform source discovery once on startup and won't recognize pods scheduled on the node after startup. This tool would be more useful it was able to monitor dynamic sources in user namespaces. thanks