Closed manish-panwar closed 5 years ago
I can send you the generated file as well if you guys want.
Thanks for reaching out. The issue here is that the template depends on consul service registration of self while it's launching and before it's marked as healthy. While zk service containers are launching and yet to be marked as healthy, {{range $i, $services := service "zk"}}
is an empty set!
If I launch another service first and wait for it to be healthy, the environment variables will be set correctly. Here, I tested registering the example redis-cache
example, see how all environment variables are set correctly except for ZOO_SELF_SERVICE
:
The scripts to reproduce and observation:
$ nomad job init --short
Example job file written to example.nomad
$ nomad job run ./example.nomad
==> Monitoring evaluation "d86a8c32"
Evaluation triggered by job "example"
Allocation "e92a9788" created: node "3b248445", group "cache"
Evaluation status changed: "pending" -> "complete"
==> Evaluation "d86a8c32" finished with status "complete"
$ nomad job run ./zk.hcl
==> Monitoring evaluation "a1647501"
Evaluation triggered by job "zk"
Allocation "38fbb42a" created: node "3b248445", group "zk"
Evaluation status changed: "pending" -> "complete"
==> Evaluation "a1647501" finished with status "complete"
$ nomad alloc logs 38fbb42a
>>>>>>>>>>>>>>> FROM ENV_VARS
ZOO_SIMPLE_RANGE=a
ZOO_OTHER_SERVICE=127.0.0.1
ZOO_SIMPLE_VAR=127.0.0.1
>>>>>>>>>>>>>>> FROM ENV_FILE
# test variable lookup
ZOO_SIMPLE_VAR="127.0.0.1"
# test range assignment
ZOO_SIMPLE_RANGE="a"
# test range over other services
ZOO_OTHER_SERVICE="127.0.0.1"
# test range over self services
The job file is:
job "zk" {
region = "global"
datacenters = ["dc1"]
type = "service"
group "zk" {
count = 1
task "zk" {
driver = "docker"
template {
data = <<EOH
# test variable lookup
{{ $node_ip := env "NOMAD_IP_zk_port" }}ZOO_SIMPLE_VAR="{{ $node_ip }}"
# test range assignment
{{ range $i, $letters := "a" | split " " }}ZOO_SIMPLE_RANGE="{{ $letters }}"{{end}}
# test range over other services
{{ range $i, $services := service "redis-cache" }}ZOO_OTHER_SERVICE="{{ $services.Address }}"{{end}}
# test range over self services
{{range $i, $services := service "zk"}}ZOO_SELF_SERVICE="{{ $services.Address }}"{{end}}
EOH
destination = "local/zkid.env"
change_mode = "noop"
env = true
}
config {
image = "alpine:latest"
command = "/bin/sh"
args = ["-c", "echo '>>>>>>>>>>>>>>> FROM ENV_VARS'; printenv | grep ZOO_; echo; echo '>>>>>>>>>>>>>>> FROM ENV_FILE'; cat /local/zkid.env; sleep 40000"]
}
service {
name = "zk"
port = "zk_port"
address_mode = "host"
}
resources {
cpu = 1024 # 1024 Mhz
memory = 512 # 512MB
network {
port "zk_port" {}
}
}
}
}
}
@notnoop - I tested it even with hardcoded stuff (I mean {{range $i, $services := 10.10.10.10}}
) and I was still seeing the issue. Surprisingly I could see Nomad returning all nodes where zk is deployed using {{range $i, $services := service "zk"}}
even though containers aren't healthy yet - but yes, I agree that it's sort of a race condition, and Nomad may not return anything since we are looking for self. I raised this issue specifically around how env variables are loaded by Nomad, not necessarily how to do lookup. Is it possible to print some logs in Nomad where I can see what values Nomad is seeing from log file? I am wondering if some extra whitespace or something causing the issue.
I have a similar issue on nomad 0.9.4
that happens sometimes and is not easy to reproduce - the template stanza listen to consul services changes (not-self service) and update file + env vars
the file created with the correct values but nomad doesn't attach the environment variables.
template stanza for kibana service:
template {
data = <<EOH
{{ range service "elasticsearch-logs" }}
ELASTICSEARCH_URL="http://{{ .Address }}:{{ .Port }}"
{{ end }}
EOH
destination = "local/elasticsearch.env"
env = true
}
@lbachar thanks for the additional context! By design, consul template substitution only blocks evaluation on key value lookup, the range lookup returns immediately with whatever is available!
@angrycub has an example workaround that calls consul kv put
after populating the range results. https://github.com/angrycub/nomad_example_jobs/tree/master/consul-template/coordination
If the kabana template is setup with the default change_mode
(restart
), it should restart every time the consul template values change, updating the env. If that's the case, we should open a new issue.
Thanks, Lang
@langmartin thanks for the quick response.
on nomad 0.8.6 we see the values re-rendered according to consul service health check.
our restart stanza has change_mode
set to restart
(default value) but the consul-template
is working well here and render the file local/elasticsearch.env
but doesn't set them as env var.
if I ssh to the docker I can see the file is rendered correctly with the service ip & port.
I tried to find similar issues here and I found those: https://github.com/hashicorp/nomad/issues/3498 https://github.com/hashicorp/nomad/pull/3529
Looks like it was fixed before using that function: MergeTemplateEnv
which I couldn't find this method in the codebase anymore.
Let me know if it's better to open a new issue instead of this one.
@lbachar Hey, sorry for the very slow response! It seems to me that #6112 is an open version of your template issue.
@langmartin Hi, indeed. I will follow #6112 Did you manage to reproduce the issue?
@lbachar , yes, this issue has been reproduced and is being addressed.
This used to work on nomad 0.9.0 but after update to 0.9.5 I get the same results: the file is generated with the service data (from range
), but the process does not see it. However in the allocation log I see that the process was restarted due to template change.
@Fuco1 , please follow #6112
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.
Nomad version
Nomad v0.9.1 (4b2bdbd9ab68a27b10c2ee781cceaaf62e114399)
Operating system and Environment details
Photon OS
Issue
If we are generating the environment variables in template dynamically based of some logic then Nomad isn't attaching the env variables to task. If we provide hardcoded values the env variables values are passed to docker container properly.
Reproduction steps
Following don't work - it generates the variable in key=value format.
This works (mind the hardcoded value):
Job file (if appropriate)
Nomad Client logs (if appropriate)
Nomad Server logs (if appropriate)