gulducat
commented
1 month ago Hey there @thetoothpick, thanks for the report!
Seems like there are two (presumably related) topics here:
- Stuck allocations
- Weird deployments
Before tackling the strange deployment behavior, I'd like to get a better sense of the state of these stuck, GC-resistant allocs, and their evaluations (allocs get GC'd with their evals - they're tightly connected).
Are you able to run these commands against one such stuck alloc, and provide the output (sanitized as needed)?
nomad alloc status -json <alloc id>
# get the EvalID from that ^
nomad eval status -json <eval id>With that, we may be able to 1) understand how they might've got stuck, and how to delete them, so the strange deployments don't occur, but also 2) perhaps I can get a test environment into the weird state to troubleshoot the deployment behavior.
If you can easily trigger one of the strange deployments, getting a eval status -json of the deployment evaluation would be great too.
In general, looking at nomad eval list and the full status of any that aren't complete might be interesting to see.
thetoothpick
Hello! We're having an issue when upgrading from Nomad 1.6.9 to Nomad 1.8.1. It seems to be caused by failed allocations that were never garbage collected (sometimes from months ago). We think we have a workaround to make our upgrade less painful, but it's still very involved.
We would like to 1) submit a bug report for the behavior, and 2) check if there's a better workaround to remove the stuck "failed" allocations in Nomad 1.8.1.
Nomad version
Nomad 1.6.9 / 1.8.1
Operating system and Environment details
Linux
Issue
We were running on Nomad 1.6.9 for some time, and upgraded directly to 1.8.1 in development environments. We're still running 1.6.9 in production.
In 1.6.9, it seems like the cluster has collected "failed" allocations and would not garbage collect them. Sometimes the allocations stick around even after its client was gone from the cluster. When we updated to 1.8.1 for the first time in development, any new deploy of a job that had these phantom failed allocations would attempt to "replace" the allocation on deployment. It took a while (and a second cluster upgrade) to figure out the mechanics of what was going on, but I think it's as follows:
nomad system gc, even though it has been weeks / months since they failed, and the client has been removed.nomad system gc.countallocations are launched, and old allocations are stopped.count(orcanary) allocations, all "extra" allocations are shut down, leaving the correct count.It seems like the root issue is that Nomad didn't record that they were replaced already, and so it tries to replace them the next chance it gets.
Our system relies on updating a number of jobs in parallel / sequentially, so having to spawn tons of extra allocations and keep failing deployments / restarting the deploy controller is a huge pain.
Right now, our current plan for upgrading production is to upgrade to 1.8.1, and trigger two evaluations for each job with failed allocations, then run a
system gc.We'd like to see if there is an easier way to get rid of these stuck failed allocations. Is there a way to forcibly remove the allocations?
I've poured over the documentation and tried everything GC-related I could find - any direct allocation GC fails because the clients are gone, and
nomad system gcis ineffective.Example
In this job,
count = 2andupdate.canary = 2. There were 4 stuck failed allocations.Screenshots
Nomad Server logs (if appropriate)
I watched the Nomad 1.6.9 server logs via the GUI monitor during a
system gc, nothing relevant was printed.Nomad CLI logs