Feature Request: Support key-value attributes for services

[rafikk]

In addition to free-form tags for services, it would be incredibly useful to have key-value attributes with client and library support.

[cbarbour]

I'm interested in this as well. As an alternative, we are considering embedding key/value pairs in tags. But this approach doesn't seem very clean.

[ryanuber]

This is a planned feature (there are a few issues logged which mention it, mostly part of #294). The main problem is that for backwards compatibility we will need to support k/v representation as well as the original flat form. It's on the roadmap, just hasn't been prioritized yet.

In the meantime, I would suggest following something like RFC1464, "Storing Arbitrary Attributes in DNS", which was originally intended for use with DNS TXT records, but solves the same problem. I know it doesn't solve your use cases fully, but for now that or JSON is the best we have.

[amochtar]

@slackpad This is the issue I was talking about. It's related to https://github.com/hashicorp/consul/issues/697.

There are also a couple of issues on it on registrator: https://github.com/gliderlabs/registrator/issues/52 / https://github.com/gliderlabs/registrator/pull/242

[Multiply]

Any update on this? We are going to need this functionality very soon, so if you want a PR for it, we might put some time into it.

[armon]

To provide some color, one of the larger ticket items on the roadmap is a V2 API. There are lots of little things we'd like to correct about V1, including adding K/V metadata to all objects (nodes, services, checks, etc).

We could just add this in places to V1, but we'd rather do it in a top-down redesign of V2. This is both to minimize our duplicated effort, and to make it more compelling to migrate to V2 instead of having to support both versions indefinitely.

Unfortunately, I don't have concrete timelines currently. Consul 0.6 is a monster release, and we cannot increase its scope any further. Consul 0.7 will follow more quickly and be a much smaller release focused on minor features to make it easier to operate and more reliable. This is one of the potential tent poles for a Consul 0.8 however.

Hope that provides some insight into our thinking! Today, the best way to simulate this is to just make use of a convention in the KV store. e.g. "service/FOO/meta/KEY" This way you can use something like Consul-Template to do a lookaside into the KV store.

[CyrilPeponnet]

:+1: For now I'm using tags to pass custom attributes. This is not recommended but I had no other choice.

[dweinstein]

:+1: I'd like to use attributes instead of tags for some things

[jmgpeeters]

Yes, this would be a brilliant feature for me as well.

[primalmotion]

:+1:

[samber]

+1

[CpuID]

+1

[hackley]

+1

[rylarson]

+1

[flozano]

+1

[winpat]

+1

[eli007s]

+1

[abhayvatsa]

+1

[cabrinoob]

Is this feature will be included in the 0.7? It will be a very good improvement used jointly with consul-template to have more flexibility in the generation of my HAProxy configuration file. (possibility to pass path-prefix, names ...etc from registrator to generate ACL dynamically)

Thank you anyway for this wonderful Consul ;)

[jhickson]

@armon Is this still a target for 0.8 as you mentioned it might be above?

[rokka-n]

This will allow better integration with load-balancers and reverse-proxy services via consul.

While KV implementation is excellent, completeness gap between service's metadata and KV's metadata makes consul integration not as fun as it should be.

[eyalstoler]

:+1: Still a game changer feature for us :smile:

[hwatts]

For the people wanting this for use with consul-template, we've worked around this by dropping a json object in as a tag, then using the parseJSON function in consul-template to access the attributes like paths, timeouts etc. Seems to work pretty well...

[djedi-bot]

@hwatts That's an interesting approach. How would you compare storing those data as KV entries with parsing them on "client" side? Do the templates still manageable?

[hwatts]

It doesn't make our templates any less manageable, although I think consul-template syntax is pretty unmanageable anyway ;)

We did consider using KVs but as we're using registrator as a bridge between docker and consul, it's a lot easier for us to stick to using tags on services, otherwise we'd be splitting this data into two locations and have to write custom code to manage the KVs on container startup.

Each microservice just has a simple json file in the project root that defines its paths/metadata etc. which we then write to an environment variable using registrator's convention and it does its magic and makes it a tag

[cbarbour]

I ran into similar problems. I originally tried solving the problem by using join and split to filter the tags and capture key value pairs.

Ultimately, we ended up writing Consul service watch handlers in Python. Overall, I'm extremely happy with that approach; it allows template generation using Jinja, complex processing of input, and complex application of the data.

IMO, consul-template is excellent for straight-forward cases. If you need really complex logic, consider writing watch handlers instead.

That said, in some cases the best approach was to simply pass the data to a CFM tool. The benefit there is that you can take advantage of existing logic for handling complex tasks. E.g. it's fairly easy to pass Consul data into Puppet to generate logical volumes and filesystems. Doing the same with Consul template or a watch handler is basically re-inventing the wheel.

[dustinbrown]

Is there any progress on this? I love the idea and would consume it immediately.

[danielsnider]

+1

[perrymanuk]

+1

[georglehmannfinalsoft]

+1

[farahfa]

It's been 2+ years! Woah!

[pierresouchay]

That would be great to implement such feature, people are using tags ad nauseam using hacks to encode it, but it would be very effective to be able to store those metadata.

And using the KV itself is not a solution as we have to implement cleanup once the instance of service is disapearing

[weiwei04]

This will solve our use case, we'd like to use this feature to store some per instance configuration(like nginx upstream weight). Looking forward to see this feature released @slackpad :)

[pierresouchay]

I just started a very basic implementation that needs tests and refinements here (not yet ready for a PR): https://github.com/pierresouchay/consul/tree/service_metadata

My idea of implementation is just a kind of KV for services (no search for now) because :

• we need to bind those metadata to the lifecycle of service (KV is not an option since it would require cleanups once the service has been removed)
• tags are not enough and are not scalable (while it might be possible to encode using base64 the values, this is very annoying) and would make searches in /catalog/services tainted by those values as well as far bigger
• Support for hierarchical structure is probably not needed (something similar to node meta is enough)

Currently my implementation is just a POC and does not supports hierarchy (just a map of key value), here is an example of request on /v1/catalog/service/my-app-http :

[
    {
        "ID": "92865e59-bba9-cd1b-b5ec-6a82aa3581f6",
        "Node": "C02Q3L7GG8WP",
        "Address": "192.168.3.10",
        "Datacenter": "dc1",
        "TaggedAddresses": {
            "lan": "192.168.3.10",
            "wan": "192.168.3.10"
        },
        "NodeMeta": {
            "consul-network-segment": ""
        },
        "ServiceID": "my-app-http",
        "ServiceName": "my-app-http",
        "ServiceTags": [
            "http",
            "canary",
            "swagger",
            "wants_lb",
            "wants_https"
        ],
        "ServiceAddress": "",
        "ServiceMeta": {
            "start_date": "2018-02-06T03:45:23.126",
            "api.version": "4",
            "api.version.min": "3",
            "http_chroot": "/my/appm",
            "lb-weight": "10",
            "swagger_path": "/api/swagger.json"
        },
        "ServicePort": 8081,
        "ServiceEnableTagOverride": false,
        "CreateIndex": 98,
        "ModifyIndex": 98
    }
]

Such example show how it could allow handling chroot on a proxy, handling versioning of APIs, load-balancer weights, and discovery of sub services (here a swagger path).

The main advantage would be to carry data within the service itself and not re-inventing yet another binding to KV or an external dependency.

What do you think ?

Do some of you have other ideas ?

@slackpad Do you think such PR could be merged after few refinements?

[chrsoo]

@pierresouchay I would like to be able to define different service types, e.g. an event source for a given type of entity. Of course we can use ad-hoc metadata for this but it would be good if you could roll your own service definitions that require specific keys when registering the service.

But I guess this goes beyond this particular feature request...

Regarding your POC I would personally prefer support for hierarchical metadata structures as this would facilitate more complex service types.

[pierresouchay]

@chrsoo I did think about this for a while while writing the code, but the issue of adding a schema to the metadata is a complex schema: in that case, you need versioning (because you want your schema to evolve right?) => which is a very complex problematic in itself (see for instance how the problem is not really solved by RDBMs).

Adding depth or types to the value is definitely doable , but I am not sure if it does not create other kind of issues in its own (for instance, readers do have to check whether a value is a String or an Integer since no schema is protecting apps to put garbage anyway) without having a schema.

I though that using the same exact model as Node Meta Data (it has the exact same limitations and validation rules as Node Meta) is probably a good way to have it merged quickly by Consul maintainers. The main advantage of linking this data to the service in itself is to have the same life-cycle than the service itself and do not create complex naming conventions and to create garbage collecting jobs in the KV to achieve similar results (this is especially interesting if you have short lived services as we do for some of our usages)

Still, if you want to put structured documents within the data, it is still possible to put some serialized JSON/YAML/XML within the metadata (we are doing this in our systems in some parts of our Consul's K/V)

I really think this is a very useful addition for the community since many people I know did express similar need in various companies for such feature.

Regards

[chrsoo]

@pierresouchay Yes I appreciate the difficulties in adding typed services and including a schema to enforce them, so best leave this for another day.

The NodeMeta seems to group information about the node on which the service runs/should run on but here we are talking about information on the service itself so why would we have a common parent field ServiceMeta?

I don't really want to get into a discussion on the difference between service properties and metadata but would it not make sense to allow e.g. a custom ApiVersion to be added at the same level as Address or ServiceID? This is at least how I would model a description of a service given a carte blanche...

Consider the headers of an HTTP request where the spec allows any header to be added as long as it follows the header format and does not conflict with any of the headers that are already defined for HTTP.

By analogy you could argue that Consul should allow any field for a service as long as it follows whatever rules we have defined for service fields. Given that these service fields seem to be any valid JSON type... well we then get back to supporting hierarchy of values similar to the KV store.

Now, this is all theoretical and I agree that there is a value in something that works now as opposed to perhaps never, so please don't be discouraged by the feedback above!

PS It used to be that custom HTTP headers should be prefixed with X- to distinguish them formal headers but this is now discouraged given that popular custom headers such as X-Forwarded-For are becoming a defacto standard and the former X- requirement thus becomes a liability...

[pierresouchay]

@chrsoo Hello Christopher,

To me global service metadata can already be achieved using simple data in KV. In our company, we are already using a namespace for that:

/kv/service-data/ can contain any data which is not specific to an instance of the service. Example: team responsible for the service, public DNS name for the service...

We are here trying to solve the issue with data specific to an instance. The issue being that for a given service, several instances might have different properties (version being the most evident because service do not get deployed at the same version at the same millisecond).

We already could perform some kind of namespacing to have metadata per instance, for instance by using /kv/service-data// and it could work, but it would be more complicated to do, especially regarding cleanup of data for old instances. The main advantage of this PR is to simply have the data attached to the instance itself.

On our side, we plan to use this internally for several use-cases:

• startTime of service (to allow to progressively send traffic to instance in order to allow the instance to warm up)
• version (and allow progressive deployment of new version and route only a few % of trafic to new version)
• weight (we are currently achieving similar results with tags, but it pollutes and increase greatly the size of /catalog/services)
• Paths to describe APIs for discovery (ex: where can I find the swagger descriptor for this API) => since versions and code do change, some of these paths may be different from version 1 to version 2

Regards

[chrsoo]

@pierresouchay Hi Pierre,

Yes you are right it does make sense to distinguish between instance and global level metadata.

Maybe you should call the field of your pending PR InstanceMeta to make it clear that we are not talking about global level metadata for the service? This had at least me confused until now...

However, what I see as central to this feature request is to obviate the need to maintain global service metadata in the KV store in addition to normal service registration. I want to avoid developing a mechanism that versions this metadata and associates the right instance with the right version.

From a client perspective it makes more sense that I provide the generic, global metadata when I register the service. This would of course mean that multiple instances of the same service would make multiple, identical entries unless there is a new version of the service.

To avoid duplication you could imagine that Consul versions global metadata documents. When retrieving a Service Definition the correct version of the global metadata would be returned together with a version number of the service.

I would call this global metadata Service Descriptor and use the field Descriptor together with associated DescriptorVersion in the Service Definition. The Service Descriptor would be a custom, arbitrary JSON (YAML) document that works in a similar way as the KV-store but that is versioned and tied to the lifecycle of the Service Definition.

[pearkes]

I think we can call this done in #3881 and available in 1.0.7 thanks to @pierresouchay! I think there may be additional features requested in this thread we can consider as improvements for the feature, please open separate issues for that.

[kadaan]

Any reason this isn't exposed in the following? https://github.com/hashicorp/consul/blob/a67d27c7564e450cd85af3e0e278a1a4c0822a79/agent/config/config.go#L317

[preetapan]

@kadaan good catch. This is a consequence of us having two different structs for service definitions (one for the HTTP API and one for service files loaded by the agent). #3881 missed adding that to the latter https://github.com/hashicorp/consul/blob/a67d27c7564e450cd85af3e0e278a1a4c0822a79/agent/config/config.go#L317. I've created https://github.com/hashicorp/consul/issues/4045 for our next release to fix this.

[pierresouchay]

@preetapan @kadaan Shame on me (Actually, we almost only register services using the HTTP API, so I did not test this case, only HTTP in https://github.com/hashicorp/consul/pull/3881), here is a fix: https://github.com/hashicorp/consul/pull/4047