search

raintank / legacy-kubernetes-app

Grafana App for Kubernetes
Apache License 2.0
77 stars 12 forks source link

Cannot use Kubernetes DNS as URL #45

Open Coolfeather2 opened 7 years ago

Coolfeather2 commented 7 years ago 
time="2017-10-16T11:45:46Z" level=warning msg="Task failed" _block=spin _module=scheduler-task consecutive failure limit=10 consecutive failures=6 error="rpc error: code = 2 desc = Error Connecting to graphite at grafana-backend.grafana:2003. Error: dial tcp: lookup grafana-backend.grafana on 169.254.169.254:53: no such host" task-id=dda70bc3-d1ec-4282-9c83-eb34fa8da251 task-name=Task-dda70bc3-d1ec-4282-9c83-eb34fa8da251

time="2017-10-16T11:45:56Z" level=error msg="error with publisher job" _module=scheduler-job block=run error="rpc error: code = 2 desc = Error Connecting to graphite at grafana-backend.grafana:2003. Error: dial tcp: lookup grafana-backend.grafana on 169.254.169.254:53: no such host" job-type=publisher plugin-config=map[port:{2003} prefix_tags:{} prefix:{snap.cluster-1.gke-cluster-1-default-pool-5a6d0c6d-f8f4} server:{grafana-backend.grafana}] plugin-name=graphite plugin-version=-1

time="2017-10-16T11:45:56Z" level=warning msg="Publish job failed" _block=submit-publish-job _module=scheduler-workflow parent-node-type=collector publish-name=graphite publish-version=-1 task-id=dda70bc3-d1ec-4282-9c83-eb34fa8da251 task-name=Task-dda70bc3-d1ec-4282-9c83-eb34fa8da251

My container and node dashboards have no data also no cluster disk usage on cluster dashboard

Coolfeather2 commented 7 years ago

changing the graphite write address to the service instead of dns name fixed the issue, but would still rather use dns for better deployment practice if it is possible

woodsaj commented 7 years ago

In order to be able to collect certain metrics, the agent runs with "hostNetwork: true". Due to using the hostNetwork , the pod running the agent will use the DNS servers defined in the hosts /etc/resolv.conf And this typically doesnt list the address of the k8s internal DNS servers.

woodsaj commented 7 years ago

This is a DUP of https://github.com/grafana/kubernetes-app/issues/38

woodsaj commented 7 years ago

However, looks like https://github.com/kubernetes/kubernetes/issues/17406 has been resolved so we can probably address this.

Though as it would restrict the app to support k8s 1.6 or newer, i am not sure we should right now.

Coolfeather2 commented 7 years ago

Okay, thanks for addressing this