Open aws-taylor opened 1 year ago
I believe the following block in getDefaultWorkers()
will cause it to have the desired effect:
if (workers == 0) {
workers = getNumberOfNeuronCores();
}
From: lxning @.> Reply-To: awslabs/multi-model-server @.> Date: Wednesday, August 24, 2022 at 12:15 PM To: awslabs/multi-model-server @.> Cc: "Goodhart, Taylor" @.>, Mention @.***> Subject: Re: [awslabs/multi-model-server] Add support for detecting the number of available Neuron cores (PR #1002)
@aws-taylorhttps://github.com/aws-taylor This PR make MMS get the number of Neuron Cores, but don't see it is used in MMS internally. Will you have another PR to address it?
— Reply to this email directly, view it on GitHubhttps://github.com/awslabs/multi-model-server/pull/1002#issuecomment-1226135196, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANYNIBQKSB272JDEDFIIR7DV2ZYBRANCNFSM57MQNFLA. You are receiving this because you were mentioned.Message ID: @.***>
Before or while filing an issue please feel free to join our slack channel to get in touch with development team, ask questions, find out what's cooking and more!
Issue #, if available:
Description of changes:
Similar to GPUs, when running on AWS Inf1 instance types, automatically detect the number of available Neuron cores and set the number of workers accordingly.
Testing done:
Ran tests on inf1, logs correctly show 16 neuron cores on an inf1.6xlarge.
To run CI tests on your changes refer README.md
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.