search

HubSpot / Singularity

Scheduler (HTTP API and webapp) for running Mesos tasks—long running processes, one-off tasks, and scheduled jobs. #hubspot-open-source
http://getsingularity.com/
Apache License 2.0
822 stars 188 forks source link

Cleanup old decommissioned agents #2253

Closed WH77 closed 2 years ago

WH77 commented 2 years ago

Assuming that if an agent is decommissioned (not stuck decommissioning) on startup and there aren't resources associated with it in the mesos master, it should be safe to clean it up the same way as dead/missing agents.

cc - @ssalinas

ssalinas commented 2 years ago

I don't know that this is actually safe. I think we also need the condition that we have not received offers from it in that amount of time as well. Currently, if an agent is still active with the mesos master, but decommissioned for us, we are continually receiving offers. So, if we were to delete it from our agents list -> get a new offer, we will actually end up adding it back as active. There is a maintenance mode in mesos master (old issue https://github.com/HubSpot/Singularity/issues/948) that we could integrate with to stop offers coming in when decomissioned

ssalinas commented 2 years ago

🚢