“tail -f” logs for a job for all instances

tolitius commented 3 years ago

Proposal

there is kubetail which enables you to aggregate (tail/follow) logs from multiple pods into one stream. This is the same as running "kubectl logs -f " but for multiple pods. … but it is for kubernetes.

nomad logs -tail -f -job [job-name]

tails logs of a single, random instance of a job which is only “somewhat” useful.

yes, logs from multiple instances can be/are shipped to ELK/splunk, etc… but they are available on all nomad instances, and while some ssh mockery with a poor user experience can be done, it is just that: not great.

we can do better.

nomad knows about the mounts OR about the logs that are available in stdout.

the proposal is to enable nomad logs to tail logs from all instances for a given app: similar to kubetail.
something like this:

nomad logs -tail -f -job [job-name] --all-instances

to see the logs in the terminal from all instances of a job.

this conversation has started on hashicorp discuss where @jrasell recommended to create an issue

Use-cases

any time logs need to be inspected (i.e. most of the time) logs from all instances for an app are needed.
a log from a single, random instance is rarely useful.

Attempted Solutions

I currently do somm like this, but it is a bit limited since it does not differentiate logs per host, does not colorize it, etc..

echo "hit ctrl-c to stop"
sleep 0.5
PIDS=""
for host in ${SERVERS[*]}
do
   ssh -tt $USER@$host "echo; echo $host; echo '----------------------------------->'; tail -qf $LOG_FILE" &
   PIDS="$PIDS $!"
done

trap 'kill $PIDS' SIGINT

wait

shoenig commented 3 years ago

Hi @tolitius, this is a great suggestion. Smaller quality of life improvements like this tend to be overlooked when we're planning out work for the Nomad roadmap. If this is something folks would really find useful, please definitely leave a :+1:

tolitius commented 1 year ago

👉 in case anyone has a need to tail / grep logs from all allocations for a job in one place feel free to use and/or contribute to https://github.com/tolitius/entail

$ entail hubble
hit ctrl-c to stop
-------------------------------------------------------------------------------------
looking at logs for       "hubble" app
logs from                 /alloc/logs/

                          allocation | version | node client
-------------------------------------------------------------------------------------
fffced31-0168-4b63-1e58-5f62d05a0857 |   v1.42 | nomad-dev-client-i-076921e2a56a65500
eba351b6-a1f1-3983-bdcb-6e7b8c9f27bd |   v1.42 | nomad-dev-client-i-014af3afd328a7a30
38e93fd6-2183-ecae-80c6-7ea75370d193 |   v1.42 | nomad-dev-client-i-0e5a4c7134c7fe2c5
-------------------------------------------------------------------------------------
2022-11-24T05:17:41,473 [pool-1-thread-1] INFO  ...
2022-11-24T06:37:43,680 [async-dispatch-5] INFO  ...
2022-11-24T06:37:43,764 [async-dispatch-6] INFO  ...
2022-11-24T06:37:43,880 [async-dispatch-7] INFO  ...

maxadamo commented 1 month ago

👉 in case anyone has a need to tail / grep logs from all allocations for a job in one place feel free to use and/or contribute to https://github.com/tolitius/entail

a livesaver! It would be nice if it had colours

hashicorp / nomad