search

bsdpot / nomad-pot-driver

Nomad task driver for launching freebsd jails.
Apache License 2.0
84 stars 14 forks source link

pots left behind in case job is dying #19

Closed grembo closed 1 year ago

grembo commented 4 years ago

In case the process of a pot dies (e.g. killall nginx) the task is restarted automatically by nomad as expected, but the pot is left behind (pot list). When doing a graceful stop of a job (nomad stop job) this isn't the case.

pizzamig commented 4 years ago

There is already a workaround for this, even if not fully documented. The pots left behind can be automatically destroyed via pot prune. nomad launched pot are marked as prunable, meaning that, once stopped, they can be removed. In minipot there is a cronjob to run pot prune periodically

grembo commented 4 years ago

There is already a workaround for this, even if not fully documented. The pots left behind can be automatically destroyed via pot prune. nomad launched pot are marked as prunable, meaning that, once stopped, they can be removed. In minipot there is a cronjob to run pot prune periodically

Is the pot marked prunable before or after launch? I'm wondering if there is a possibility for bad timing leading to accidental destruction of a pot that is in the middle of getting provisioned if pot prune is run from cron.

pizzamig commented 4 years ago

The mechanism is implemented in 2 steps to avoid the situation you described. There are 2 attributes: prunable and to-be-pruned. The attribute prunable can be set by the user and automatically set by pot prepare (this command has been designed to interact with nomad). The attribute to-be-pruned is set by pot start if prunable is true. pot prune will remove pots that are not running and that have the attribute prunableand to-be-pruned set to true.

grembo commented 4 years ago

It might make sense to integrate this feature into the pot driver (maybe configurable?).

pizzamig commented 4 years ago

There is a garbage collection feature in nomad and the idea is to run pot prune in that moment. It's currently not working and we need to investigate the cause. An alternative idea is to run pot prune after the pot is stopped.

grembo commented 2 years ago

@pizzamig I think the changes in pizzamig/pot#200 and trivago/nomad-pot-driver#30 might actually solve this one as well.

einsiedlerkrebs commented 1 year ago

comment moved to #35 (as new issue)

grembo commented 1 year ago

I think this has been overcome by events