nomad alloc exec options passing works incorrectly

[valodzka]

With any container that has ls. This works as expected:

nomad alloc exec 6659bfac /bin/ls -l

This prints nomad help instead ls help:

nomad alloc exec 6659bfac /bin/ls -h

Expected behaviour: nomad should pass all options after container id to program.

[DerekStrickland]

Hi @valodzka. Thanks for using Nomad and for reporting this issue.

I've confirmed the behavior you are seeing on main but I believe this is as expected.

The Nomad documentation provides a couple of examples of how to issue remote commands. Here's one that I found useful.

nomad alloc exec <alloc_id> /bin/sh -c 'cat /output | sort | uniq -c | sort -rn | head -n 5'

From that example you can see that with exec you are starting a new remote shell, and any command you want that shell to run must be passed inside single quotes or an inline heredoc in the -c flag. Using that approach, I was able to print the ls help text with the following command.

nomad alloc exec <alloc_id> /bin/sh -c 'ls --help'

Hope that helps! I'm closing this issue, but feel free to reopen if you think there is still something to be addressed.

Side note, while investigating this issue, I noticed that the short version of the help flag (-h) didn't print anything. I execed into the running container and noticed that this was true even from inside the container. After looking at the help output I realized -h has a different purpose in ls.

-h, --human-readable       with -l and/or -s, print human readable sizes
                              (e.g., 1K 234M 2G)

So make sure to use --help!

[valodzka]

Hi @DerekStrickland, thank you for your response. While this behavior might be as expected and there are workarounds I think it's inconsistent and because of this it's bad design.

• ls -l andls --help should work in same way, both pass arguments or both don't.
• Other shell utilities work in different way (sudo, flock, xargs, ...), for example: flock --help prints flock help and flock /tmp ls --help prints ls help, so nomad behavior is also inconsistent here.

feel free to reopen if you think there is still something to be addressed.

I can't reopen bugs. It's of course not a critical UX bug so it's up to you to decide to reopen and fix this or keep it as is.

[DerekStrickland]

Hi @valodzka

You are absolutely correct. There is an inconsistency here. I'll take a look. Thank you for being persistent.

[DerekStrickland]

The challenge here is that the upstream mitchellh/cli library is interpreting this as the general help flag for the CLI. It intercepts the input, and prints the command help text without ever running the AllocExecCommand. Research will have to be done to see if there is a way to special case this behavior. For now there is a workaround by using -c as described in the thread above.

[valodzka]

One more argument for fix is that -c workaround is not always possible, for example with distroless docker images..

[tgross]

Leaving a note here that the usual GNU getopts behavior of stopping argument parsing after -- doesn't work either:

$ nomad alloc exec b94 -- df -h
OCI runtime exec failed: exec failed: unable to start container process: exec: "--": executable file not found in $PATH: unknown

[lgfa29]

Noting here that there is previous discussion about supporting -- in https://github.com/hashicorp/nomad/issues/6527#issuecomment-901611716