Ability to limit AutoML resource using (amount of parallel threads)

[80LevelElf]

The current AutoML is not good to use when you have trained a lot of models at the same time in the cloud. If you have some amount of pods in your Kubernetes cluster it doesn't matter how many AutoML experiments you execute at the same time. 4 experiments or 1 experiment at the same time use 100% of CPU (It brokes health checks and so on)

Low-level API of trainers (like FastForestBinaryTrainer) has options like NumberOfThreads and some other trainer-specific options you can use to handle the workload.

Is it able to add something like this to AutoML API?

But the most brilliant solution is some sort of smart property like ResourceUsingRatio which can be from 0.0 to 1.0 ResourceUsingRatio = 1.0 means the experiment use the maximum of potential resources (mainly CPU) it needs or the machine has.

[michaelgsharp]

@80LevelElf thanks for your suggestion, this is something that would be good to address. We are currently in the process of updating our AutoML implementation, so this is something that we can look at during this time.

@JakeRadMSFT @LittleLittleCloud thoughts?

[LittleLittleCloud]

In model builder and mlnet cli, users can limit on the number of thread one experiment can use via env:MLNET_MAX_THREAD, we could probably do the same via AutoML experiment setting

[michaelgsharp]

@LittleLittleCloud will that environment variable work with AutoML just as is? As a potential workaround for now:?

[LittleLittleCloud]

Nope,, it's only available in modelbuilder/cli

[michaelgsharp]

Yeah, I just checked and the main ML.NET framework itself doesn't know anything about that..

With the re-work going on with AutoML, is this something that could be included pretty easily with it?

[LittleLittleCloud]

Yup, just need to expose it through AutoML experiment setting and override the default value when creating trainer