influxdb Database creation failed

[jmreicha]

I am running Kubernetes v0.14.2, CoreOS on AWS platform and following the instructions to turn up Heapster using the kubectl create -f deploy/kube-config/influxdb/ command.

Pods, RCs and services seem to get created okay but I noticed the following in the log grafana pod logs (via kubectl log monitoring-heapster-controller-26873).

2015-04-11T00:24:53.414429145Z + EXTRA_ARGS=
2015-04-11T00:24:53.414485300Z + '[' '!' -z '""' ']'
2015-04-11T00:24:53.414500150Z + EXTRA_ARGS='""'
2015-04-11T00:24:53.414597602Z + '[' '!' -z 10.244.0.1 ']'
2015-04-11T00:24:53.414597602Z + EXTRA_ARGS='--kubernetes_master 10.244.0.1:80 ""'
2015-04-11T00:24:53.414620734Z + HEAPSTER=/usr/bin/heapster
2015-04-11T00:24:53.414738668Z + case $SINK in
2015-04-11T00:24:53.414762507Z + HEAPSTER='/usr/bin/heapster --sink influxdb'
2015-04-11T00:24:53.414771068Z + '[' '!' -z 10.244.0.1 ']'
2015-04-11T00:24:53.414778741Z + INFLUXDB_ADDRESS=
2015-04-11T00:24:53.414797022Z + '[' '!' -z 10.244.36.227 ']'
2015-04-11T00:24:53.414826064Z + INFLUXDB_ADDRESS=10.244.36.227:80
2015-04-11T00:24:53.414854342Z + /usr/bin/heapster --sink influxdb --sink_influxdb_host 10.244.36.227:80 --kubernetes_master 10.244.0.1:80 '""'
2015-04-11T00:24:53.422331365Z I0411 00:24:53.421526       7 heapster.go:45] /usr/bin/heapster --sink influxdb --sink_influxdb_host 10.244.36.227:80 --kubernetes_master 10.244.0.1:80 ""
2015-04-11T00:24:53.422331365Z I0411 00:24:53.422095       7 heapster.go:46] Heapster version 0.10.0
2015-04-11T00:24:53.422331365Z I0411 00:24:53.422161       7 kube.go:237] Using Kubernetes client with master "http://10.244.0.1:80" and version v1beta1
2015-04-11T00:24:53.422331365Z I0411 00:24:53.422174       7 kube.go:238] Using kubelet port "10250"
2015-04-11T00:24:53.422331365Z I0411 00:24:53.422198       7 driver.go:164] Using influxdb on host "10.244.36.227:80" with database "k8s"
2015-04-11T00:24:53.525288101Z E0411 00:24:53.524627       7 driver.go:175] Database creation failed: Post http://10.244.36.227:80/db?u=root&p=root: net/http: transport closed before response was received. Retrying after 30 seconds
2015-04-11T00:25:23.525907630Z E0411 00:25:23.525863       7 driver.go:175] Database creation failed: Post http://10.244.36.227:80/db?u=root&p=root: read tcp 10.244.36.227:80: connection reset by peer. Retrying after 30 seconds
2015-04-11T00:26:17.574399625Z E0411 00:26:17.574356       7 driver.go:175] Database creation failed: Post http://10.244.36.227:80/db?u=root&p=root: read tcp 10.244.36.227:80: connection reset by peer. Retrying after 30 seconds
2015-04-11T00:27:11.624016034Z E0411 00:27:11.623943       7 driver.go:175] Database creation failed: Post http://10.244.36.227:80/db?u=root&p=root: read tcp 10.244.36.227:80: connection reset by peer. Retrying after 30 seconds
2015-04-11T00:28:05.673956790Z E0411 00:28:05.673903       7 driver.go:175] Database creation failed: Post http://10.244.36.227:80/db?u=root&p=root: read tcp 10.244.36.227:80: connection reset by peer. Retrying after 30 seconds
2015-04-11T00:28:59.721990128Z E0411 00:28:59.721918       7 driver.go:175] Database creation failed: Post http://10.244.36.227:80/db?u=root&p=root: read tcp 10.244.36.227:80: connection reset by peer. Retrying after 30 seconds

I am also unsure from the readme what address I should be visiting to access the Grafana GUI? I see the reference to https://<master_iP>/api/v1beta1/proxy/services/monitoring-grafana is that correct? Or will it be one of the node IP's?

When I run a curl to the above master address I get the following Error: 'dial tcp 10.244.103.11:8080: no route to host'.

[vishh]

This happens when the influxdb pod is not online. Heapster and InfluxDB are created in parallel, and heapster waits for InfluxDB to be online before proceeding with data collection. By default, you can access Grafana at 'https:///api/ v1beta1/proxy/services/monitoring-grafana/' . If you add a 'hostport' option to InfluxDB pod, it will be accessible via the node IPs.

On Fri, Apr 10, 2015 at 5:37 PM, Josh Reichardt notifications@github.com wrote:

I am running Kubernetes v0.14.2, following the instructions to turn up Heapster using the kubectl create -f deploy/kube-config/influxdb/ command.

Pods, RCs and services seem to get created okay but I noticed the following in the log grafana pod logs (via kubectl og monitoring-heapster-controller-26873).

2015-04-11T00:24:53.414429145Z + EXTRA_ARGS= 2015-04-11T00:24:53.414485300Z + '[' '!' -z '""' ']' 2015-04-11T00:24:53.414500150Z + EXTRA_ARGS='""' 2015-04-11T00:24:53.414597602Z + '[' '!' -z 10.244.0.1 ']' 2015-04-11T00:24:53.414597602Z + EXTRA_ARGS='--kubernetes_master 10.244.0.1:80 ""' 2015-04-11T00:24:53.414620734Z + HEAPSTER=/usr/bin/heapster 2015-04-11T00:24:53.414738668Z + case $SINK in 2015-04-11T00:24:53.414762507Z + HEAPSTER='/usr/bin/heapster --sink influxdb' 2015-04-11T00:24:53.414771068Z + '[' '!' -z 10.244.0.1 ']' 2015-04-11T00:24:53.414778741Z + INFLUXDB_ADDRESS= 2015-04-11T00:24:53.414797022Z + '[' '!' -z 10.244.36.227 ']' 2015-04-11T00:24:53.414826064Z + INFLUXDB_ADDRESS=10.244.36.227:80 2015-04-11T00:24:53.414854342Z + /usr/bin/heapster --sink influxdb --sink_influxdb_host 10.244.36.227:80 --kubernetes_master 10.244.0.1:80 '""' 2015-04-11T00:24:53.422331365Z I0411 00:24:53.421526 7 heapster.go:45] /usr/bin/heapster --sink influxdb --sink_influxdb_host 10.244.36.227:80 --kubernetes_master 10.244.0.1:80 "" 2015-04-11T00:24:53.422331365Z I0411 00:24:53.422095 7 heapster.go:46] Heapster version 0.10.0 2015-04-11T00:24:53.422331365Z I0411 00:24:53.422161 7 kube.go:237] Using Kubernetes client with master "http://10.244.0.1:80" and version v1beta1 2015-04-11T00:24:53.422331365Z I0411 00:24:53.422174 7 kube.go:238] Using kubelet port "10250" 2015-04-11T00:24:53.422331365Z I0411 00:24:53.422198 7 driver.go:164] Using influxdb on host "10.244.36.227:80" with database "k8s" 2015-04-11T00:24:53.525288101Z E0411 00:24:53.524627 7 driver.go:175] Database creation failed: Post http://10.244.36.227:80/db?u=root&p=root: net/http: transport closed before response was received. Retrying after 30 seconds 2015-04-11T00:25:23.525907630Z E0411 00:25:23.525863 7 driver.go:175] Database creation failed: Post http://10.244.36.227:80/db?u=root&p=root: read tcp 10.244.36.227:80: connection reset by peer. Retrying after 30 seconds 2015-04-11T00:26:17.574399625Z E0411 00:26:17.574356 7 driver.go:175] Database creation failed: Post http://10.244.36.227:80/db?u=root&p=root: read tcp 10.244.36.227:80: connection reset by peer. Retrying after 30 seconds 2015-04-11T00:27:11.624016034Z E0411 00:27:11.623943 7 driver.go:175] Database creation failed: Post http://10.244.36.227:80/db?u=root&p=root: read tcp 10.244.36.227:80: connection reset by peer. Retrying after 30 seconds 2015-04-11T00:28:05.673956790Z E0411 00:28:05.673903 7 driver.go:175] Database creation failed: Post http://10.244.36.227:80/db?u=root&p=root: read tcp 10.244.36.227:80: connection reset by peer. Retrying after 30 seconds 2015-04-11T00:28:59.721990128Z E0411 00:28:59.721918 7 driver.go:175] Database creation failed: Post http://10.244.36.227:80/db?u=root&p=root: read tcp 10.244.36.227:80: connection reset by peer. Retrying after 30 seconds

I am also unsure from the readme what address I should be visiting to access the Grafana GUI? I see the reference to https://

/api/v1beta1/proxy/services/monitoring-grafana is that correct? Or will it be one of the node IP's? — Reply to this email directly or view it on GitHub https://github.com/GoogleCloudPlatform/heapster/issues/231.

[jmreicha]

@vishh is there an example somewhere that I can follow? I was having issues browsing to the url you pointed out.

[vishh]

What issues are you facing? Did you replace the '' portion in the url with your kubernetes master IP? Are you running the kubernetes api-server in a custom port?

On Mon, Apr 13, 2015 at 7:19 PM, Josh Reichardt notifications@github.com wrote:

@vishh https://github.com/vishh is there an example of that somewhere? I was having issues browsing to the url you pointed out.

— Reply to this email directly or view it on GitHub https://github.com/GoogleCloudPlatform/heapster/issues/231#issuecomment-92563294 .

[jmreicha]

When I run a curl to the above master address I get the following Error: 'dial tcp 10.244.103.11:8080: no route to host'.

I am running the Kubernetes master on port 8080.

[vishh]

Don't you have to provide master auth?

On Tue, Apr 14, 2015 at 8:02 AM, Josh Reichardt notifications@github.com wrote:

When I run a curl to the above master address I get the following Error: 'dial tcp 10.244.103.11:8080: no route to host'.

I am running the Kubernetes master on port 8080.

— Reply to this email directly or view it on GitHub https://github.com/GoogleCloudPlatform/heapster/issues/231#issuecomment-92893430 .

[jmreicha]

That's what I'm confused about. Maybe I am not following a correct Kubernetes setup?

I am using the CoreOS config guide - https://github.com/GoogleCloudPlatform/kubernetes/tree/master/docs/getting-started-guides/coreos/cloud-configs

[vishh]

It looks like the apiserver doesn't have any auth configured. Are you able to access the api-server? By that I mean access `http://

:/`.

[vishh]

Are there any firewall rules that you need to configure to access the master?

On Tue, Apr 14, 2015 at 12:54 PM, Vishnu Kannan vishnuk@google.com wrote:

It looks like the apiserver doesn't have any auth configured. Are you able to access the api-server? By that I mean access `http://

:/`.

[jmreicha]

No firewall rules, right now internally it is pretty wide open. I can curl http://master-ip:8080 okay.

[vishh]

This sounds like a kubernetes issue. Can you raise an issue against kubernetes or may be reach out to some of the CoreOS users on IRC (google-containers on freenode)?

On Tue, Apr 14, 2015 at 1:05 PM, Josh Reichardt notifications@github.com wrote:

No firewall rules, right now internally it is pretty wide open. I can curl http://: okay.

— Reply to this email directly or view it on GitHub https://github.com/GoogleCloudPlatform/heapster/issues/231#issuecomment-93043483 .

[jmreicha]

Maybe I will just hold off until Kubernetes stabilizes a little bit :/

[vishh]

Ah! May be you can connect with @jonlangemak who maintains a kubernetes on CoreOS setup with really awesome guides!

On Tue, Apr 14, 2015 at 2:05 PM, Josh Reichardt notifications@github.com wrote:

Maybe I will just hold off until Kubernetes stabilizes a little bit :/

— Reply to this email directly or view it on GitHub https://github.com/GoogleCloudPlatform/heapster/issues/231#issuecomment-93060954 .

[jonlangemak]

My setup is on bare metal CentOS7 but the pod deployment for the cluster add-on should be the same. I've been using branch/release 0.13 though since I had issues with Kubernetes in anything after that. I havent tried .15 yet though. Let me know if you want the link for my config based on .13

[jmreicha]

Yeah a link to your config would be great.

[jonlangemak]

I did a write up on how I deployed Heapster here -> http://www.dasblinkenlichten.com/installing-cadvisor-and-heapster-on-bare-metal-kubernetes/ It's a little dated (but updated) since the initial install relied on cadvisor running as a manifest on each host before it got baked into the kubelet.

As far as the configs for replications controllers and services go you can checkout my github page here -> https://github.com/jonlangemak/saltstackv2 The premise of that repo is to use SaltStack to deploy a baremetal cluster so there's a lot of templating in the configs but there's a very limited amount (if any) in the heapster YAML files so it should be pretty straight forward. See this page -> https://github.com/jonlangemak/saltstackv2/tree/master/salt/pods/heapster specifically for the Heapster replication controller and service definitions.

Would love to hear how you're testing goes. Like I said, a lot changed after .13 and I couldnt get the API server service proxy to work in .14 so I stuck with .13. Im hoping it's fixed in .15 but I havent checked yet. A lot also changed after .13 in regards to how things like etcd are deployed and what services run where etc.

And Im rambling. Feel free to shoot me a email with further questions or comments
jon at interubernet dot com

[mwielgus]

Obsolete, retries were added.