search

elijah / chef-prometheus

Chef cookbook for Prometheus. The open source service monitoring system and time-series database.
Apache License 2.0
52 stars 93 forks source link

Plans for Job Configuration? #20

Closed ewr closed 9 years ago

ewr commented 9 years ago

What are your plans re: job specifications? I see the service discovery question in #6, but I'm thinking more in terms of a LWRP that wrapper cookbooks can call.

I'm still trying to evaluate if Prometheus is going to be what I'm looking for, but I'm glad to help out on implementing that sort of thing over the next few days. I just want to make sure I'm generally pointed in the same direction you'd like to see the cookbook go.

rayrod2030 commented 9 years ago

I think ultimately a LWRP is where we need to go in order to allow people to implement their own discovery mechanism and use this cookbook with a wrapper cookbook as you have mentioned. I was thinking of just doing something chef discovery specific using search but even that feature would benefit from using a LWRP to generate the job template so yes it's well worth it to write a job config LWRP.

ewr commented 9 years ago

Am I correct in not seeing any support for prometheus to read multiple config files? If so, that's going to be the complicated bit in terms of a job LWRP—you need to make sure all the LWRP calls get made before you write the single config file.

A similar case would be the iptables-ng cookbook. The rule and chain providers write out individual files, and then delayed notify a ruby block that generates the final output file at the end:

https://github.com/chr4-cookbooks/iptables-ng/blob/01f79041e03aef17fe2212a856970ca928792f09/providers/rule.rb#L66-L67

I'll take a shot at doing something in that direction.

rayrod2030 commented 9 years ago

Yeah good point another cookbook that I thought about that does this is the heavy water graphite cookbook: https://github.com/hw-cookbooks/graphite. Multiple resources merge to build one or a few graphite config files.

rayrod2030 commented 9 years ago

Also note that the core prometheus project itself is working on adding support for difference discovery services in addition to their DNS based discovery support but it would certainly be useful to have in the cookbook as well.

rayrod2030 commented 9 years ago

Some more details on the Accumulator Pattern that's used in the graphite cookbook to take multiple resources and merge them into a file or template: https://github.com/kisoku/chef-accumulator

ewr commented 9 years ago

Neat. I hadn't seen chef-accumulator.

I forgot about the DNS SRV support. We run Consul, which supports DNS SRV, so that might work for part of what I need (what nodes / ports should I query for a given service?).

I'm working on the client cookbook side for a little bit, but will definitely circle back around to this piece by tomorrow morning.

ewr commented 9 years ago

Here's where I'm at in case any of this is useful...

Yesterday, I built out a little cookbook for client nodes. It currently has a node resource that supports grabbing a binary node-exporter, installing it and setting up an Upstart service:

https://github.com/scpr-cookbooks/prometheus-client

I also built a wrapper cookbook that uses that resource to set up the node exporter and register into a Consul service:

https://github.com/scpr-cookbooks/scpr-prometheus-client

On the other side, this wrapper cookbook sets up the server and then sets up our node-exporter job from Consul:

https://github.com/scpr-cookbooks/scpr-prometheus

rayrod2030 commented 9 years ago

I meant to reply to this thread yesterday. I actually have tried using the dns based service discovery built into prometheus using consul/SRV records/dnsmasq and could not get it to work. Would be curious to see if you have better luck.

ewr commented 9 years ago

It seems to work for me...

screen shot 2015-03-13 at 11 30 25 am

You can check out the scpr-consul cookbook to see how I'm setting up dnsmasq, etc.

ewr commented 9 years ago

Just noticed my prometheus port is wrong there... Need to take the extra colon out of the spec for that job in the default recipe.

rayrod2030 commented 9 years ago

Ah cool I have to revisit what I had set up before.

ewr commented 9 years ago

Just FYI since you mentioned using Consul: I'm playing with building an exporter to monitor service health (and eventually keys that should have active locks). Just got the first working pass this morning:

https://github.com/ewr/consul_exporter

Example results:

screen shot 2015-03-18 at 9 19 57 am

rayrod2030 commented 9 years ago

Very cool. Thanks!

On Wed, Mar 18, 2015 at 9:21 AM, Eric Richardson notifications@github.com wrote:

Just FYI since you mentioned using Consul: I'm playing with building an exporter to monitor service health (and eventually keys that should have active locks). Just got the first working pass this morning:

https://github.com/ewr/consul_exporter

Example results:

[image: screen shot 2015-03-18 at 9 19 57 am] https://cloud.githubusercontent.com/assets/204350/6709476/031aa264-cd50-11e4-9808-3583c1722655.png

— Reply to this email directly or view it on GitHub https://github.com/rayrod2030/chef-prometheus/issues/20#issuecomment-82967765 .

ewr commented 9 years ago

Closed by #22.