Gossip arbitrary metadata about services

[jaym]

The example use-case I'm thinking for this is gluing together something like prometheus and the services that need monitoring. Binding to service groups seems tedious. If services could gossip metadata like "I'm a thing that needs to be monitored", the prometheus could render a config based off things based on that metadata.

[eeyun]

We've decided this is a feature that we want to implement. That being said, we still need to have a discussion around what the implementation of this feature might look like.

It will likely require quite a bit of thought to get the right shape. So for now we're going to keep the S-needs-discussion label and determine the appropriate path going forward.

[baumanj]

Confirmed valid in v0.57

[adamhjk]

I'm :-1: on having this be arbitrary. I think the pattern we are looking for is one that is declarative, not one that involves having a random lookup.

What about a reverse binding? Where rather than declaring who fulfils that binding when prometheus starts, you get it from the other service? --bind-to promethus.prod:services or something like that, where you get the aggregate of all bound services?

[jaym]

I think this all came about because the first thing I tried didn't work:

hab svc load jaym/prometheus --channel unstable --bind targets:hello-service.default --bind targets:world-service.default

It happily accepted that, but it didn't merge the service groups and the rendered config only had one of the services. I checked and the latest version of hab still behaves that way. From what I understand, this means that the core plan for prometheus is not really that useful if you have more than one service group. To make it work, you'd need to create a new plan, and replace the config for a section for every service group you have.

I did a little demo here of what I thought would solve my problem: https://github.com/jaym/habitat-selector-binds/blob/master/.studiorc. The idea is that when starting something like prometheus, I want to tell it to bind targets(services) to things that implement a certain interface. In the case of the demo, that was things with metrics-http-port. But this could be expanded to say things that implement the prometheus interface and things that are in a certain group (like the .default part of foo.default), etc. I think this approach allows services to glue themselves together a bit more.

In the example of metrics, the thing that jumps out to me is that if you wanted to try swapping in a different metrics poller thing, it would require updating all the bind-to for each running service. That being said, if you gave me --bind-to, I would use it over creating another prometheus service package with a place to bind each of my service groups.

[adamhjk]

This is one of the questions we're still figuring out in habitat land. Does it even make sense to deploy "prometheus", with all the generic configuration done from that package? It's unclear to me that is easier than creating a prometheus-yourstuff service that depends on prometheus, and just uses prometheus' native configuration syntax to do the right thing. Chef certainly went the "cookbook is the interface" path, and we landed with 100% of Chef's complexity + 100% of the underlying softwares, and you really needed to be good at both to be effective. With habitat, we've chosen the opposite direction - that it's actually easier to just write the configuration file like the upstream would have told you to, only marked up with configuration.

That said - I think having both a bind-to and perhaps some kind of merge binding is interesting. But I'm pretty sure that, faced with making prometheus generic to all uses vs making a prometheus plan that is easy to wrap with specifics, I would chose the wrap option (because it has a much lower concept count, and is easy to extend with minimal prometheus and minimal hab knowledge).

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.