Open finwo opened 2 years ago
Hi @finwo thanks for the report, nomados
looks like an interesting project!
The root cause here is likely the lack of a df
executable used for finding the mount point of the data directory
https://github.com/hashicorp/nomad/blob/v1.2.6/client/fingerprint/storage_unix.go#L32
If there's a better way to get the same information without resorting to exec
, we can definitely look into incorporating it.
Sounds reasonable @danishprakash. Of course the best way to know would be to try it :slightly_smiling_face:
@shoenig Just tested if adding a df
binary worked.
When using the df
binary from ubase, the error remained.
Using the df
binary from my host system (void linux), nomad started up and registered succesfully as a client with my previously-existing server.
I can confirm this issue when using busybox's df
. It appears that the functionality of busybox's df
is not equivalent in some way to coreutils df
.
Is someone internally already working on it or should I give it a go with statfs/statx?
Hey @finwo feel free to work on this!
hackety hack config in client stanza:
options {
"fingerprint.denylist" = "storage"
}
got my agent starting.
for some reason, fingerprinting only failed when starting under systemd on eL8. didn't bother digging deeper.
I'm not sure if this will help your case but i'm building this using busybox (for the moment for debugging) and noticed that the call to df was actually failing due to os.Exec not resolving it from the path.
(Note that in busybox echo $PATH
does show /sbin:/usr/sbin:/bin:/usr/bin
but this isn't set as the path for other processes (and wont be seen when running env
so you need to explicitly export it in the shell so it is available to the env)
in any case, this does look to work as expected when using busybox's df
implementation (also note that I've called mount -t tmpfs tmpfs /tmp
(among other things) as the code calls something like df -kP /tmp/NomadClient9823174204
(using nomad agent -dev
this if using the provided /etc/config/init.json
to use the state_dir
specified based in the original config (so /var/lib/nomad will need to be mounted as appropriate to resolve in the df call/storage fingerprinting)
Nomad version
1.2.6, freshly downloaded from https://releases.hashicorp.com/nomad/1.2.6/nomad_1.2.6_linux_amd64.zip
Operating system and Environment details
Minimal linux kernel with custom init, built & run using
make test
from https://github.com/finwo/nomados/Start command:
/sbin/nomad agent -dev -log-level=TRACE -config=/etc/nomad/init.json
/sbin/nomad agent -log-level=TRACE -config=/etc/nomad/init.json
Config:
Issue
agent: error starting agent: error="client setup failed: fingerprinting failed: failed to determine disk space for /var/lib/nomad/data/alloc: failed to determine mount point"
Nomad throws an error early during startup as shown above, stating it can't determine the available disk space or mount point, when running in a minimal environment on ramdisk (as in the linked repository).
Reproduction steps
Clone the linked repository, run
make test
in the root of the repsitory, wait for it all to build, start and crashExpected Result
A virtual machine running as a nomad client which joins the server that's already running in my network
Actual Result
After getting an IP and sarting nomad, nomad throws an error (
agent: error starting agent: error="client setup failed: fingerprinting failed: failed to determine disk space for /var/lib/nomad/data/alloc: failed to determine mount point"
) stating it can't determine the mount point for it's alloc directory.The problem arises both with and without the
-dev
flag, although without the flag it depends on/sbin/ip
to be present before the error is thrown.Job file (if appropriate)
n/a
Nomad Server logs (if appropriate)
n/a
Nomad Client logs (if appropriate)