search

HubSpot / Singularity

Scheduler (HTTP API and webapp) for running Mesos tasks—long running processes, one-off tasks, and scheduled jobs. #hubspot-open-source
http://getsingularity.com/
Apache License 2.0
823 stars 188 forks source link

singularity holding offers for long time #1532

Closed bydga closed 7 years ago

bydga commented 7 years ago

Hey, we just noticed (we are running singularity on one cluster together with marathon) that sometimes marathon takes ages to deploy new app. After inspecting whats wrong, I found out that singularity keeps all offers and they are visible in the "outstanding offers" tab in the mesos-ui: http://prntscr.com/f5r2pr.

Seems like the only way marathon can deploy something is, when the offers times-out, mesos sends to both frameworks new ones and marathon catches his breath.. and then Singularity eats everything again..

Is there some setting I'm missing or is it an expected behavior?

mesos: 1.2.0 singularity 0.14.1 marathon: 1.4.2

PtrTeixeira commented 7 years ago

See also PR 1402; resolved by PR #1529.

ssalinas commented 7 years ago

This is the same as the issue referenced in https://github.com/HubSpot/Singularity/pull/1402#issuecomment-299815613 . You can set cacheOffers: false in your config yaml to return to the previous behavior. https://github.com/HubSpot/Singularity/pull/1529 adds the docs for this and will turn the offer cache off by default

ssalinas commented 7 years ago

@PtrTeixeira beat me to it ;)

bydga commented 7 years ago

oh, thanks! Somehow i searched only issues and missed the discussions in the PRs, thanks for pointing out :)