Run reload service but config do not change

[lamhaison]

Docker: 1.7.1 Consul: 0.6.4 - latest Consul-template version: 0.12.0

I have to update image servicea for two docker containers when update code for staging env. I perform task:

• delete container
• create container with new image

But I have two docker engine and each host run one docker nginx(nginx and auto reload config nginx when change of consult using consul-template service) on this host. I divide my Stag env to two environment children, stag-1(run container docker-serviceA), stag-2(run container docker-serviceB) due to HA for service A.

I have filter when update configuration from consul when have change

• nginx on docker ngine stag1 only add config host:port of docker-serviceA on this host -stag1
• nginx on docker nginx stag2 only add ocnfig host:port of docker-serviceA on this host stag2

In my process, i will deploy on stag1 and after that deploy on stag2. Because require HA (do not died both two container service A on same time).

But when i update image for container service A on stag1, nginx on docker nginx stag1 and stag2 aslo restart (Not Allow this because service will died a little time). I seem that is only configuration of nginx on stag1 change (because update image on stag1, do not on stag2), configuration of nginx on stag2 do not change. I don't understand why nginx on host on stag2 reload despite of do not any change of configuration on nginx in host on stag2

I think, good idea is have check changing of configuration before run command restart service nginx.

Thanks

[sethvargo]

Hi @lamhaison

Thank you for opening an issue. Consul Template will not reload the service unless the resulting configuration file changes. Could you please supply your debug output?

[lamhaison]

My nginx template

{{$app := env "APP_NAME"}}
{{$tag := env "TAG"}}

upstream {{printf $app}} {
    least_conn;
    {{range service $app}}
    {{ if .Tags | contains $tag }}
    server {{.Address}}:{{.Port}} max_fails=3 fail_timeout=60 weight=1;{{end}}
    {{ end }}
}

server {
    listen 80 default;

    location / {
        proxy_pass http://{{printf $app}};
    }
}

Render configuration of consul-template service

nginx-stag-1 - 10.10.11.51 - tag: node-1-stag-1
upstream tutum-nodejs-redis {
    least_conn;
    server 10.10.11.51:32770 max_fails=3 fail_timeout=60 weight=1;
    server 10.10.11.51:32771 max_fails=3 fail_timeout=60 weight=1;
}

nginx-stag-2 - 10.10.11.60 -tag: node-2-stag-2
upstream tutum-nodejs-redis {
    least_conn;
    server 10.10.11.60:32777 max_fails=3 fail_timeout=60 weight=1;
    server 10.10.11.60:32778 max_fails=3 fail_timeout=60 weight=1;
}

But when I remove or restart container service on stag-1: 10.10.11.51. consult will raise log on nginx stag-2

2016/07/15 21:13:25 [DEBUG] ("service(tutum-nodejs-redis)") **Consul returned 4 services**
2016/07/15 21:13:25 [DEBUG] ("service(tutum-nodejs-redis)") 4 services after health check status filtering
2016/07/15 21:13:25 [DEBUG] (view) "service(tutum-nodejs-redis)" no new data (index was the same)
2016/07/15 21:13:25 [DEBUG] ("service(tutum-nodejs-redis)") querying consul with &{Datacenter: AllowStale:true RequireConsistent:false WaitIndex:5314 WaitTime:1m0s Token: Near:}

2016/07/15 21:14:13 [DEBUG] ("service(tutum-nodejs-redis)") **Consul returned 3 services**
2016/07/15 21:14:13 [DEBUG] ("service(tutum-nodejs-redis)") 3 services after health check status filtering
2016/07/15 21:14:13 [INFO] (view) "service(tutum-nodejs-redis)" received data
2016/07/15 21:14:13 [DEBUG] (view) "service(tutum-nodejs-redis)" starting fetch
2016/07/15 21:14:13 [DEBUG] ("service(tutum-nodejs-redis)") querying consul with &{Datacenter: AllowStale:true RequireConsistent:false WaitIndex:5403 WaitTime:1m0s Token: Near:}
2016/07/15 21:14:13 [DEBUG] (runner) receiving dependency "service(tutum-nodejs-redis)"
2016/07/15 21:14:13 [INFO] (runner) running
2016/07/15 21:14:13 [DEBUG] (runner) checking template /templates/default.ctmpl
2016/07/15 21:14:13 [DEBUG] (runner) checking ctemplate &{Source:/templates/default.ctmpl Destination:/etc/nginx/conf.d/default.conf Command:service nginx reload CommandTimeout:30s Perms:-rw-r--r-- Backup:false LeftDelim: RightDelim: Wait:0xc8200117b0}
2016/07/15 21:14:13 [DEBUG] (runner) wouldRender: true, didRender: true
2016/07/15 21:14:13 [DEBUG] (runner) appending command: service nginx reload
2016/07/15 21:14:13 [INFO] (runner) diffing and updating dependencies
2016/07/15 21:14:13 [DEBUG] (runner) "service(tutum-nodejs-redis)" is still needed
2016/07/15 21:14:13 [DEBUG] (runner) running command: `service nginx reload`, timeout: 30s
Reloading nginx: nginx.
2016/07/15 21:14:13 [INFO] (runner) watching 1 dependencies
2016/07/15 21:15:15 [DEBUG] ("service(tutum-nodejs-redis)") Consul returned 3 services
2016/07/15 21:15:15 [DEBUG] ("service(tutum-nodejs-redis)") 3 services after health check status filtering
2016/07/15 21:15:15 [DEBUG] (view) "service(tutum-nodejs-redis)" no new data (index was the same)
2016/07/15 21:15:15 [DEBUG] ("service(tutum-nodejs-redis)") querying consul with &{Datacenter: AllowStale:true RequireConsistent:false WaitIndex:5403 WaitTime:1m0s Token: Near:}

It only care about change total service, but do not care whether configuration of nginx have changed or not. So that reload sometime is not good, It will down my service when i deploy code

[sethvargo]

Hi @lamhaison

This is the expected behavior. You are rendering a configuration that points to the container on 10.10.11.51, so when you restart that container, the health check will fail and that node will be removed from the list of services. That changes the nginx configuration, which forces a reload of nginx.

[lamhaison]

Remove container on 10.10.11.51. The configuration of nginx on 10.10.11.51 will change(because it include configuration container on this). But nginx on 10.10.11.60 also reload(only include configuration of containter on 10.10.11.60. So that it will not change). This is not good behavior

[jippi]

are you sure there are no extra newlines in 10.10.11.60 when 10.10.11.51 leaves in the upstream section?

[lamhaison]

I make sure, because I have filter result of consul list. nginx on 10.10.11.60 only get container on this server 10.10.11.60. Only have new empty line add to this configuration file, do not change main configuration

upstream tutum-nodejs-redis {
    least_conn;

    server 10.10.11.60:32777 max_fails=3 fail_timeout=60 weight=1;

    server 10.10.11.60:32778 max_fails=3 fail_timeout=60 weight=1;

}

[jippi]

the output above also does not match your original output from nginx.

new lines are also a file change, afaik a checksum is used to detect file changes

please see Text and spaces at https://golang.org/pkg/text/template/ on how to make sure no extra newlines are injected

possible example to trim the white space

upstream {{ printf $app }} {
    least_conn;
    {{ range service $app }}
    {{- if .Tags | contains $tag -}}server {{.Address}}:{{.Port}} max_fails=3 fail_timeout=60 weight=1;
    {{- end -}}
    {{ end }}
}

[lamhaison]

Yes, this is good template. My nginx service restart like as expected result that I wan (Do not restart when do not change configuration). Thanks you so much, consul-template is very good service for auto scale.