HubSpot / Singularity

Scheduler (HTTP API and webapp) for running Mesos tasks—long running processes, one-off tasks, and scheduled jobs. #hubspot-open-source
http://getsingularity.com/
Apache License 2.0
823 stars 188 forks source link

How to use largeScaleDownAcknowledged flag? #2305

Closed yuriy-filatov closed 2 years ago

yuriy-filatov commented 2 years ago

Currently we're stuck with

Cannot scale down by more than 10 instances at a time without explicit acknowledgement (set the largeScaleDownAcknowledged field in the request)

I've tried to add largeScaleDownAcknowledged=true to s9y request section in the job's yaml -- but no luck. Can you please explain how to use this toggle? Thanks.

ssalinas commented 2 years ago

The flag is present as a query param on the scale request. See https://github.com/HubSpot/Singularity/blob/3b120a92c304c659cd894b1f790f8b10378db991/SingularityService/src/main/java/com/hubspot/singularity/resources/RequestResource.java#L1786 adn the deploy resource, if executing a deploy that changes scale. It is also present as a checkbox in the UI if scaling via UI, or as an argument on the Singularity java client if scaling via api

yuriy-filatov commented 2 years ago

Thanks @ssalinas Is there a global toggle in s9y server config I can bump to override defaults, i.e. set default to 50 instances or allow largeScaleDown? Usecase is -- we have a running deployment with 50 job instances and trying to 'update' it through our ci/cd -- but get the error above.

ssalinas commented 2 years ago

If you want to effectively turn off the check, you can set maxScaleDownWithoutAcknowledgement at the base of the yaml to a much larger number. Default is 10. Alternatively, just acknowledge by setting the value to true on the scale/deploy api calls

https://github.com/HubSpot/Singularity/blob/0b420cb6231b61f017f18a08a564e852f4395eae/SingularityService/src/main/java/com/hubspot/singularity/config/SingularityConfiguration.java#L240

yuriy-filatov commented 2 years ago

Thanks, we'll give it a try.

yuriy-filatov commented 2 years ago

Works for us with global override, thanks.