Closed grembo closed 1 year ago
There is already a workaround for this, even if not fully documented.
The pot
s left behind can be automatically destroyed via pot prune
. nomad launched pot
are marked as prunable, meaning that, once stopped, they can be removed.
In minipot
there is a cronjob to run pot prune
periodically
There is already a workaround for this, even if not fully documented. The
pot
s left behind can be automatically destroyed viapot prune
. nomad launchedpot
are marked as prunable, meaning that, once stopped, they can be removed. Inminipot
there is a cronjob to runpot prune
periodically
Is the pot marked prunable before or after launch? I'm wondering if there is a possibility for bad timing leading to accidental destruction of a pot that is in the middle of getting provisioned if pot prune
is run from cron.
The mechanism is implemented in 2 steps to avoid the situation you described.
There are 2 attributes: prunable and to-be-pruned.
The attribute prunable can be set by the user and automatically set by pot prepare
(this command has been designed to interact with nomad).
The attribute to-be-pruned is set by pot start
if prunable is true.
pot prune
will remove pot
s that are not running and that have the attribute prunable
and to-be-pruned
set to true.
It might make sense to integrate this feature into the pot driver (maybe configurable?).
There is a garbage collection feature in nomad and the idea is to run pot prune
in that moment.
It's currently not working and we need to investigate the cause.
An alternative idea is to run pot prune
after the pot is stopped.
@pizzamig I think the changes in pizzamig/pot#200 and trivago/nomad-pot-driver#30 might actually solve this one as well.
comment moved to #35 (as new issue)
I think this has been overcome by events
In case the process of a pot dies (e.g.
killall nginx
) the task is restarted automatically by nomad as expected, but the pot is left behind (pot list
). When doing a graceful stop of a job (nomad stop job
) this isn't the case.