CDCgov / cfa_azure

Apache License 2.0
8 stars 2 forks source link

use an AutoScale formula by default to discourage use of dedicated nodes in pool #51

Closed arik-shurygin closed 3 months ago

arik-shurygin commented 3 months ago

I recently uncovered an issue where a pool would fail to shut down and incur charges for each dedicated node.

I think it would be more cost effective to use a conservative autoscaling formula by default that will automatically close down nodes after use, rather than relying on users to remember to shut their pools down.

For example, a user could use/create pool and client.set_scaling() would automatically be called with defaults unless otherwise specified.

When on the web UI there are optional template auto-scale formulas to pick from, maybe start by using one of those? Here is one I found:

// In this example, the pool size is adjusted based on the number of tasks in the queue.
// Note that both comments and line breaks are acceptable in formula strings.

// Get pending tasks for the past 15 minutes.
$samples = $ActiveTasks.GetSamplePercent(TimeInterval_Minute * 15);
// If we have fewer than 70 percent data points, we use the last sample point, otherwise we use the maximum of last sample point and the history average.
$tasks = $samples < 70 ? max(0, $ActiveTasks.GetSample(1)) :
max( $ActiveTasks.GetSample(1), avg($ActiveTasks.GetSample(TimeInterval_Minute * 15)));
// If number of pending tasks is not 0, set targetVM to pending tasks, otherwise half of current dedicated.
$targetVMs = $tasks > 0 ? $tasks : max(0, $TargetDedicatedNodes / 2);
// The pool size is capped at 20, if target VM value is more than that, set it to 20. This value should be adjusted according to your use case.
cappedPoolSize = 20;
$TargetDedicatedNodes = max(0, min($targetVMs, cappedPoolSize));
// Set node deallocation mode - keep nodes active only until tasks finish
$NodeDeallocationOption = taskcompletion;
ryanraaschCDC commented 3 months ago

I like this idea! I might try to work in an option to use a default autoscale formula but allow the user to specify the max number of nodes. I'll see what I can do.