Open dnk8n opened 3 years ago
Part of the problem, although I am not sure how to fix it. It looks like nomad is listening on the wrong IP address.
vagrant@nomad1:~$ nc -zv 127.0.0.1 4646
localhost [127.0.0.1] 4646 (?) : Connection refused
vagrant@nomad1:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:8d:c0:4d brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:fe8d:c04d/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:37:fd:17 brd ff:ff:ff:ff:ff:ff
inet 10.1.42.70/24 brd 10.1.42.255 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:fe37:fd17/64 scope link
valid_lft forever preferred_lft forever
vagrant@nomad1:~$ nc -zv 10.0.2.15 4646
10.0.2.15: inverse host lookup failed: Unknown host
(UNKNOWN) [10.0.2.15] 4646 (?) open
vagrant@nomad1:~$ nc -zv 10.1.42.70 4646
nomad1.local [10.1.42.70] 4646 (?) : Connection refused
vagrant@nomad1:~$ wget 10.0.2.15:4646
--2020-11-19 13:04:42-- http://10.0.2.15:4646/
Connecting to 10.0.2.15:4646... connected.
HTTP request sent, awaiting response... 307 Temporary Redirect
Location: /ui/ [following]
--2020-11-19 13:04:42-- http://10.0.2.15:4646/ui/
Reusing existing connection to 10.0.2.15:4646.
HTTP request sent, awaiting response... 200 OK
Length: 1478 (1.4K) [text/html]
Saving to: ‘index.html’
index.html 100%[===============================================================================>] 1.44K --.-KB/s in 0s
2020-11-19 13:04:42 (55.9 MB/s) - ‘index.html’ saved [1478/1478]
vagrant@nomad1:~$ cat index.html
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>Nomad</title>
<meta name="description" content="">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="nomad-ui/config/environment" content="%7B%22modulePrefix%22%3A%22nomad-ui%22%2C%22environment%22%3A%22production%22%2C%22rootURL%22%3A%22%2Fui%2F%22%2C%22locationType%22%3A%22auto%22%2C%22EmberENV%22%3A%7B%22FEATURES%22%3A%7B%7D%2C%22EXTEND_PROTOTYPES%22%3A%7B%22Date%22%3Afalse%7D%2C%22_JQUERY_INTEGRATION%22%3Atrue%7D%2C%22APP%22%3A%7B%22blockingQueries%22%3Atrue%2C%22mirageScenario%22%3A%22smallCluster%22%2C%22mirageWithNamespaces%22%3Atrue%2C%22mirageWithTokens%22%3Atrue%2C%22mirageWithRegions%22%3Atrue%7D%2C%22ember-cli-mirage%22%3A%7B%22usingProxy%22%3Atrue%2C%22useDefaultPassthroughs%22%3Atrue%7D%2C%22exportApplicationGlobal%22%3Afalse%7D" />
<link rel="stylesheet" href="/ui/assets/vendor-d8602bc1ee5a13b26e7066b76b62d063.css">
<link rel="stylesheet" href="/ui/assets/nomad-ui-2ffef17efbbc4361902d6cc1d42ce2b8.css">
<link rel="icon" type="image/png" href="/ui//favicon-1c2527a7a07d130ecbafce75e4615a69.png" />
</head>
<body>
<script src="/ui/assets/vendor-44114fe8d7aaeb1fcb23a1020f2623c2.js"></script>
<script src="/ui/assets/nomad-ui-3f5aade8a94ca946d42dfb0ab70ffee0.js"></script>
<div id="ember-basic-dropdown-wormhole"></div>
</body>
</html>
I have since moved on from the example Vagrantfile (I will post my setup soon). I suffered from exactly the same issue though and some changes I made helped...
Ansible Role Variables:
nomad_bind_address: 0.0.0.0
(avoids the error in this issue... but then shows that no leader could be elected, see next line for a fix)
nomad_bootstrap_expect: 1
(Maybe 2 in this example... but it defaults to 3 erroneously, in the case where there are 3+ servers but fewer than 3 server nodes. With this correctly defined, servers and clients are found)
nomad_iface: enp0s8
(Due to the extra interface identified, nomad is advertising through the wrong IP. I made a change that fixed that (bit worried that it is hardcoded). In this example, I think the value should change to "eth1")
nomad_advertise_address: <set differently per node>
(Not sure if this was required or not but it is working with it set. E.g. in the example nomad_advertise_address for nomad1 is 10.1.42.70
In the following repo (https://github.com/dnk8n/monitored-orchestration/tree/main/dev) you can find a Vagrantfile that uses both ansible-nomad and ansible-consul to spin up a dev cluster. You might want to reconfigure the VM resources to suit you.
In it I follow some of the suggestions I make to work around the problem of this issue. I tried to get away from needing to have "nomad_bind_address" => "0.0.0.0"
. Using shell variable NOMAD_ADDR="http://{{ nomad_bind_address }}:4646" with nomad_bind_address set to the private IP of the VM does work for this issue, but causes another issue which can be seen in sudo systemctl status nomad
(It erroneously tries to connect via rpc to 127.0.0.1:4647 instead of the configured bind-address)
Not sure if all the modifications were necessary. Often had to totally destroy and re-vagrant up
in order to be sure on changes. Finally I got a clean run which works out of the box.
I tested with nomad init
output (cat example.nomad
) and uploaded it through the UI. It worked as expected.
The UI endpoints are:
nomad: http://n1.lan:4646 http://n2.lan:4646 http://n3.lan:4646
consul: http://n1.lan:8500 http://n2.lan:8500 http://n3.lan:8500