search

timescale / prometheus-postgresql-adapter

Use PostgreSQL as a remote storage database for Prometheus
Apache License 2.0
336 stars 66 forks source link

Prometheus unable to connect to adapter #56

Closed nicklinck closed 5 years ago

nicklinck commented 5 years ago

I am currently running prometheus on a kubernetes cluster (Kubernetes Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0+icp-ee", GitCommit:"9cb64de4ca4d039c35f4a29721aa5cf787648a15", GitTreeState:"clean", BuildDate:"2018-04-27T06:32:18Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"})

I created a kubernetes service so that the containers for pg_prometheus, prometheus_postgresql_adapter, and prometheus can communicate with each other. The .yaml file I used to create this is:

apiVersion: v1 kind: Service metadata: name: pg-prometheus-service   namespace: "data-collection"   labels:     collect: data spec:   selector:     collect: data   ports:   - name: pg-prometheus     protocol: TCP     port: 5432     targetPort: 5432    - name: postgresql-adapter     protocol: TCP     port: 9201     targetPort: 9201

I created the pg_prometheus deployment using the following .yaml file:

apiVersion: apps/v1 kind: Deployment metadata:   name: pg-prometheus   namespace: data-collection   labels:     collect: data spec:   selector:     matchLabels:       collect: data   replicas: 1   template:     metadata:       labels:         collect: data     spec:       containers:       - name: pg-prometheus         image: timescale/pg_prometheus:master         ports:         - containerPort: 5432         args: ["postgres", "-csynchronous_commit=off"]

and I created the adapter deployment using the following .yaml file:

apiVersion: apps/v1 kind: Deployment metadata:   name: prometheus-postgresql-adapter   namespace: data-collection   labels:     collect: data spec:   selector:     matchLabels:       collect: data   replicas: 1   template:     metadata:       labels:         collect: data     spec:       containers:       - name: prometheus-postgresql-adapter         image: timescale/prometheus-postgresql-adapter:master         ports:         - containerPort: 9201         args: ["-pg.host=pg-prometheus-service", "-pg.prometheus-log-samples"]`

when connecting to pg-prometheus I see that the database schema has been created:

postgres=# \dt List of relations Schema |      Name      | Type | Owner
--------+----------------+-------+---------- public | metrics_copy | table | postgres public | metrics_labels | table | postgres public | metrics_values | table | postgres (3 rows)

Prometheus and the node exporter have been running on the cluster as expected, I have just been trying to add on the remote_write feature. The problem that I am having is when I load prometheus (which is in the kube-system namespace). I edit the configmap for my prometheus deployment so that in the prometheus.yml file I have:

prometheus.yml: global:   scrape_interval: 1m   evaluation_interval: 1m

rule_files:   - /etc/alert-rules/.rules   - /etc/alert-rules/.yml

scrape_configs:   - job_name: prometheus     static_configs:       - targets:         - localhost:9090

remote_write:   - url: "http://pg-prometheus-service.data-collection:9201/write" ... (much more in the configuration file)

I then restart the prometheus deployment and get the following error:

level=info ts=2019-01-11T01:31:42.871609412Z caller=main.go:394 msg="Loading configuration file" filename=/etc/config/prometheus.yml level=error ts=2019-01-11T01:31:42.874975125Z caller=main.go:356 msg="Error loading config" err="couldn't load configuration (--config.file=/etc/config/prometheus.yml): url for remote_write is empty"

I have tried many different options for the url parameter with no success (getting rid of http://, not including the service namespace, using the IP address rather than the service name, not including /write...).

On my kubernetes cluster I am also running a kube-dns service to handle domain name resolutions.

Any advice on what might be going wrong would be greatly appreciated! Thanks!

niksajakovljevic commented 5 years ago

@nicklinck You should go into prometheus container, cat /etc/config/prometheus.yml and inspect the config file. You can share it here as well. Seems there might be some problem in your config map for prometheus.

nicklinck commented 5 years ago

@niksajakovljevic here is my prometheus.yml file: prometheus.yml

niksajakovljevic commented 5 years ago

@nicklinck I think that url should be within single quotes, not double. BTW there is a promtool (binary that comes with Prometheus) that can validate the config file: ./promtool check config prometheus.yml

nicklinck commented 5 years ago

@niksajakovljevic That solved it! Thank you! My remote_write was interfering with scrape_configs so I just had to move remote_write above it in the yaml file.