Query by tag

[NoumanSaleem]

I don't see it mentioned in the docs, so this would be a new feature request {{service "api.*"}}

[sethvargo]

Hi @NoumanSaleem,

You can query by tag using the .tag syntax described here. It's very similar to what you have suggested:

{{service "release.webapp@east-aws:8000"}}

This is querying Consul for the "webapp" service, with the "release" tag, in the "east-aws" datacenter, using port "8000".

[NoumanSaleem]

@sethvargo what I'm actually trying to accomplish is querying across all services by tag. In my case, I have multiple services which will be routed to by the same proxy.

users: ["api"]
blogs: ["api"]
memcached: []

Probably not a common use case

[sethvargo]

@NoumanSaleem ah. We've actually thought about this. In your example, you'll just want two loops:

{{service "users.api"}}
thing{{end}}
{{service "blogs.api"}}
thing{{end}}

Does that make sense?

[NoumanSaleem]

@sethvargo totally does, but was trying to make it a little dynamic ;) not knowing services ahead of time. I've put together an app in the meantime to watch for changes using the REST api, and write the config how I need it. Will switch to consul-template if tag only support comes in the future :)

thanks!

[sethvargo]

@armon what are your thoughts here?

[armon]

This is not currently possible given Consul's API. We index by service, not by tag. It is kind of a strange use case to lookup tags across all services.

[sielaq]

I think I understand the problem. We also have the same issue. What we try to accomplish is to get a list of all services from consul - since we don't know the service name yet, we can't make a template.

Typical use case: service1.service.consul service2.service.consul ... serviceN.service.consul

HAproxy config that allows access to all of them by port 80 (proxy layer 7) mostly for humans, human can access for service documentation like: service1.service.consul/docs

frontend http-in
    bind *:80

    acl  acl_consul hdr(host) -i consul.service.consul
    use_backend  backend_consul if acl_consul
    acl  acl_service1 hdr(host) -i service1.service.consul
    use_backend  backend_service1 if acl_service1   
#...itareate it dynamically for every new service...

backend backend_consul    
    balance leastconn
    option httpclose
    server 0_consul2_consul 10.0.0.1:8500 maxconn 32
    server 0_consul3_consul 10.0.0.2:8500 maxconn 32

backend backend_service1 
    balance leastconn
    option httpclose
    server 0_services3_consul 10.0.0.3:1234 maxconn 32
    server 0_services4_consul 10.0.0.4:4567 maxconn 32

#... continue it till serviceN...

And since services it selfs are dynamic we are force to make a "template for a template" which it not a nice soution. It would be much easier to iterate over services, as NoumanSaleem suggested - with reular expression it would be even better.

[armon]

I actually think the template for a template is a pretty decent solution for this. Would it help if there was a master "services" entry that just provided the list of all known services? Then you could iterate over that to generate the per-service blocks.

[sielaq]

For us - Yes, it would definitely help, script logic would be much simpler. I'm not sure if @NoumanSaleem would also appreciate that.

[NoumanSaleem]

Would totally help

[sethvargo]

@armon that won't work as expected because there's two phases to the compilation of a template - first we get the list of dependencies we need, then we query for them. If we have a master list of services, you wouldn't be able to use those in other queries because they would be dynamically compiled. Does that make sense?

[armon]

@sethvargo I think it would require template chaining. Basically instance A of consul-template sucks in the list of services and emits a new template that has each service block. Instance B of consul-template reads that input template, and generates the final output for HAProxy/Nginx

[williambailey]

How about some kind of byTag template function that accepts a []*Service and returns a map[string][]*Service where the map key is the service tag. This would then allow you to have dynamic configuration by having a known service name and then using the tag as the dynamic identifier.

Sticking with the api example one could have all of your relevant applications register that they provide an api service. Each of these services would then have one or more tags assigned which we can use to identify the type of api that they application is providing. In your consul-template template you could then do something like the following.

{{ range $tag, $services := service "api" | byTag }}
      # {{ $tag }} API end points.
      {{ range $services }}
          {{ .Address }}:{{ .Port }}
      {{ end }}
{{ end }}

[sethvargo]

@williambailey the problem with that approach is that Consul Template is a two-pass implementation. The first time it reads the template to figure out what services/keys to watch, then it queries Consul. So having some kind of range/loop in the template is rather difficult with the current implementation. That's why @armon suggested having multiple Consul Template instances - one that queries the master services list and renders a template that is a ctmpl, and then another Consul Template instance that consumes that instance.

[williambailey]

@sethvargo - The two-pass implementation remains unaffected with my suggestion. The service function is responsible for figuring out the service dependencies. The byTag function simply takes the result of the service function and populates a map that can be used in the template.

[sethvargo]

@NoumanSaleem does the new byTag functionality provide the functionality you need?

[sielaq]

@sethvargo, How about iterating over all services (so without knowing service name)? Is it possible already? "byTag" works for me (which is preety cool by the way) but still only for known services.

[sethvargo]

@sielaq I haven't added a way to iterate over all services yet, but that's "coming soon"(tm) :smile:

[NoumanSaleem]

@sethvargo as @sielaq said, it would be perfect if it iterated over all services. That + byTag would work for me. For now, template for a template is still working just fine. A definite step in the right direction! Thanks all.

[sethvargo]

Hello,

After some research, I do not think providing a services function is currently possible due to an upstream bug in Consul. Once that issue has been resolved, we can revisit this scenario.

[srobertson]

I have the same issue described above.

I'm running haproxy to share a bunch of micro services on port 80. Problem is I don't know what they're names are ahead of time.

[sethvargo]

Hi @srobertson

As you can see from the conversation and label, this is an upstream bug in Consul and Consul Template cannot query by tag until Consul supports it.

[srobertson]

Thanks for the response!

Understood, just wanted you to know the use case described is quite desireable by others.

[bryanlarsen]

AFAICT, this is now a duplicate of #77. the 'byTag' functionality has been provided, all that remains is a mechanism for querying all services, which is requested in #77.

The flags are different, though. This one is marked as "upstream bug" implying we need something from consul before this bug can proceed and #77 is marked as "enhancement" implying we don't need to wait.

[sethvargo]

@bryanlarsen it's slightly different. The participants in this issue want to query the services catalog, not the health endpoint.

[sielaq]

from version 0.5.0 services was introduced, but I'm not sure if I use it properly. this example gives me "half" empty output of consul service itself (I choosed consul sine everyone have it):

frontend http-in
    bind *:80
    {{range services}}acl acl_{{.Name}} hdr(host) -i {{.Name}}.service.consul
    use_backend backend_{{.Name}} if acl_{{.Name}}
    {{end}}
    {{range services }}backend backend_{{.Name}}
      balance leastconn
      option httpclose
    {{range service .Name "passing" }}  server {{.Name}} {{.Address}}:{{.Port}} maxconn 32
    {{end}}{{end}}

result:

frontend http-in
    bind *:80
    acl acl_consul hdr(host) -i consul.service.consul
    use_backend backend_consul if acl_consul

    backend backend_consul
      balance leastconn
      option httpclose

BUT, if I mention about service before like:

{{ $services := service "consul" }}
frontend http-in
    bind *:80
    {{range services}}acl acl_{{.Name}} hdr(host) -i {{.Name}}.service.consul
    use_backend backend_{{.Name}} if acl_{{.Name}}
    {{end}}
    {{range services }}backend backend_{{.Name}}
      balance leastconn
      option httpclose
    {{range service .Name "passing" }}  server {{.Name}} {{.Address}}:{{.Port}} maxconn 32
    {{end}}{{end}}

suddenly backeds appears:

frontend http-in
    bind *:80
    acl acl_consul hdr(host) -i consul.service.consul
    use_backend backend_consul if acl_consul

    backend backend_consul
      balance leastconn
      option httpclose
      server consul 192.168.59.103:8300 maxconn 32

So is it by design or is this just a first step and this is in your plans, and "patient I must have" :) ?

[sethvargo]

@sielaq services is different than service. services returns a list of Catalog services, but service returns a health check service object, which has significantly more information.

This is still an upstream bug in Consul, and sorry for the confusion around service vs services.

[sielaq]

@sethvargo , So does it mean that I cannot combine both of those? like:

{{range services}}        # this supposed to iterate over "Catalog Services"
  {{range service .Name}} # this supposed to iterate over "service instances" for every .Name which comes from "Catalog Services"
    {{.Address}}:{{.Port}}
  {{end}}
{{end}}

Because this is what my previous examples do, probbaly indenting is misleading... If I cannot combine both then we missunderstand what "Catalog Services" is :(

[sethvargo]

@sielaq no because of #64 :smile:

The different between catalog services and agent services is explained in the Consul API documentation.

[sielaq]

@sethvargo , Ok so we understant it properly. Catalog Service - is a global catalog - not a local agent catalog -- exacly what we need. But the combination of both "servieces" and "service" is a logically proper, both of those used "catalog/services" and "catalog/service/" just it is not working yet, due to https://github.com/hashicorp/consul-template/issues/64 ? Or this combination is logically wrong ? (I just wanna clarify - sorry for being picky :) )

EDIT: I read again about "service" and it end point is "health/service/", but still docummentation says "This is very similar to the /v1/catalog/service endpoint however, this endpoint automatically returns the status of the associated health check, as well as any system level health checks."

so my question is still valid, is it logically proper ? I understand that in the end, result of generated config can contain an empty service (where all instances are in "critical" state).

[sethvargo]

@sielaq yes - you can check the length of the services returned and handle the "zero" case (perhaps put up a maintenance page or something)

[amiorin]

Looking forward for this!

In the meanwhile I'm using monit (have a look to my answer to #109) with two invocations of template-consul. It would be nice to have a filter for local services. The goal is to have a reverse proxy routing by domain on every machine for local services. consul.service.dc1.consul is an exception because the web interface is on 8500 and not 8300 (the advertised port). This allows users to type services in the browser without the port.

Maybe consul could implement this feature instead of using nginx/haproxy/apache + consul-template. It looks to me as a killer feature for microservices architectures.

[sethvargo]

@sielaq not yet. Need to solve #64

[sielaq]

I don't wanna preempt facts :) while @sethvargo is doing an awesome job :trophy: (he actually made so many cool changes in current master branch) that I was able to print all services&instances with specific Tag (l know that this is not a final solution but just to let you know that this is possible)

{{range services}}
  {{$service:=.Name}}
  {{range .Tags}}
    {{if eq . "webapp" }}
      {{$service}}
      {{range service $service "passing" }}
        server {{.Name}} {{.Address}}:{{.Port}}
      {{end}}
    {{end}}
  {{end}}
{{end}}

In theory we can have now tags for HAproxy,Varnish,Nginx, so each realize its own services in configuration. :cool:

[jgranstrom]

I came across this as I was trying to do something similar for an API gateway. Here's what I ended up with that covered my use case. Might be of use to anyone doing something similar.

frontend http
    bind *:80
    {{range services}}{{if in .Tags "api"}}
    acl is-api-{{.Name}}-path path_beg /api/{{.Name}}
    use_backend {{.Name}} if is-api-{{.Name}}-path
    {{end}}{{end}}

{{range services}}{{if in .Tags "api"}}
backend {{.Name}}{{$services:=service .Name "passing"}}
  {{if gt (len $services) 0}}
  mode http
  option http-server-close
  reqrep ^([^\ ]*\ /)api/{{.Name}}[/]?(.*)     \1\2
  balance roundrobin{{range $services}}
  server {{.ID}} {{.Address}}:{{.Port}} check{{end}}
  {{else}}
  server dead 127.0.0.1:65535 # 503
  {{end}}
{{end}}{{end}}

[bfgoodrich]

Is there any progress on upstream consul issue #64? Would love to see this feature without the above workaround.

[ldshi]

Thanks for the great job.

No one is expecting this anymore?

I reached here because I was searching the consul api for letting me query a group of service by tag, coz I don't know the serviceId/serviceName, but I grouped the services by tag.

The only way I can see by using the current api set is: listing all the services in the dc then filter by tag myself.

So, can we give this api(globally query service by tag) a consideration?

[sethvargo]

Hi @leonard-sxy

This requires upstream apis in Consul. I would recommend opening an issue there, which is why this is tagged "upstream". Thanks! 😄

[sielaq]

@sethvargo What? I thought you have already did that (opening an issue there), or you have it in your private consul kanban queue, rotfl I would have wait forever :)

[ldshi]

@sielaq What? So u just wait there for almost 2 years???

[sielaq]

@leonard-sxy I have plenty other stuff to do :) and I can be patient sometimes :) EDIT: In parallel my colleague @magiconair has created Fabio so It took out a big part of a configuration we use to have.

[sethvargo]

@sielaq there's probably already an upstream issue, but I don't know what it is offhand.

@leonard-sxy both Consul and Consul Template are open source. We work really hard on all the projects, but we are still a small team. If you would like to see things implemented faster, you are more than welcome to contribute 😄 . Also, fabio is great @leonard-sxy ❤️

[ldshi]

@sethvargo @sielaq ic, thanks, when I am familiar enough with the whole design philosophy I will try to contribute.

[bobvanderlinden]

I ran into this issue here as well. I initially thought I was able to do something like:

{{ services ".api" }}

to get all services that have the api tag, regardless of name. The problem I'm facing is that all services have different names. I'm now considering naming all services api and tagging them with the actual name of the microservice (or adding the name to the meta). I don't know what the right way to structure consul services is yet.

[akhayyat]

Any updates on this issue? This is a pretty old one!

I have the same use case: I want to get all the services tagged to be exposed through a reverse proxy.

TBH, I'm surprised by the limited querying capabilities, given that Consul is mainly a [meta]data store, and that the data model is not that complex.

[sodabrew]

@akhayyat You can fetch all services with a particular tag using the byTag feature: https://github.com/hashicorp/consul-template#bytag

Here's an example of an haproxy configuration:

{{range $tag, $services := services | byTag}}{{ if eq $tag "public" }}{{range $services}}
backend {{.Name}}
{{range service .Name}}
   server {{.Node}} {{.Address}}:{{.Port}}{{end}}
{{end}}{{end}}{{end}}

Edit: actually this fetches all services and then filters by tag so if you have a lot of services it might end up being inefficient for your environment.

There is no API to query by tag (or query by service meta!) https://consul.io/api-docs/catalog#node-meta-1

[garry-t]

{{range $tag, $services := services | byTag}}{{ if eq $tag "public" }}{{range $services}}
backend {{.Name}}
{{range service .Name}}
   server {{.Node}} {{.Address}}:{{.Port}}{{end}}
{{end}}{{end}}{{end}}

This queries only data from primary DC, but not from all.