Multi gpu support

[waytrue17]

Description of changes: Enabling multi gpu support. It passes context information to hander functions so that model/data can be assigned to multiple gpu devices.

• To enable this feature, customer will need to change the custom handler declaration to add the context. For example, from input_fn(input_data, content_type) to input_fn(input_data, content_type, context).
• For backward compatibility, this implementation will not break existing use cases where no context gets passed e.g. input_fn(input_data, content_type) will still work.
• Tested the implementation on a 8 gpu instance with SAGEMAKER_MODEL_SERVER_WORKERS=9. The workers were assigned to different gpus as expected:

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A    396466      C   /opt/conda/bin/python3.8         1729MiB |
|    1   N/A  N/A    396465      C   /opt/conda/bin/python3.8         1729MiB |
|    1   N/A  N/A    396467      C   /opt/conda/bin/python3.8         1729MiB |
|    2   N/A  N/A    396470      C   /opt/conda/bin/python3.8         1729MiB |
|    3   N/A  N/A    396462      C   /opt/conda/bin/python3.8         1729MiB |
|    4   N/A  N/A    396469      C   /opt/conda/bin/python3.8         1729MiB |
|    5   N/A  N/A    396463      C   /opt/conda/bin/python3.8         1729MiB |
|    6   N/A  N/A    396468      C   /opt/conda/bin/python3.8         1729MiB |
|    7   N/A  N/A    396464      C   /opt/conda/bin/python3.8         1729MiB |
+-----------------------------------------------------------------------------+

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[sagemaker-bot]

AWS CodeBuild CI Report

• CodeBuild project: sagemaker-pytorch-inference-toolkit-pr
• Commit ID: 47d6f21b0eac5fa22f0f75f90732eee87def87dd
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[sagemaker-bot]

AWS CodeBuild CI Report

• CodeBuild project: sagemaker-pytorch-inference-toolkit-pr
• Commit ID: 2a0dbdab1651496f2f17ce53af49b714e6341f19
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[sagemaker-bot]

AWS CodeBuild CI Report

• CodeBuild project: sagemaker-pytorch-inference-toolkit-pr
• Commit ID: d869551cd86e2c956bb203bccb6c85617922f592
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[sagemaker-bot]

AWS CodeBuild CI Report

• CodeBuild project: sagemaker-pytorch-inference-toolkit-pr
• Commit ID: c774dd03d0555ace13f05fbf7394d57807fdde33
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[sagemaker-bot]

AWS CodeBuild CI Report

• CodeBuild project: sagemaker-pytorch-inference-toolkit-pr
• Commit ID: b5c50374b99bed0547de5265692ec067431b3127
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[sagemaker-bot]

AWS CodeBuild CI Report

• CodeBuild project: sagemaker-pytorch-inference-toolkit-pr
• Commit ID: 1ab82495518e265b2de9639fb9afb744cdb45e5d
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[sagemaker-bot]

AWS CodeBuild CI Report

• CodeBuild project: sagemaker-pytorch-inference-toolkit-pr
• Commit ID: 7fad13e07ba97b8de49b7ad1847d528fe33f774e
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[sagemaker-bot]

AWS CodeBuild CI Report

• CodeBuild project: sagemaker-pytorch-inference-toolkit-pr
• Commit ID: 2d6ce947d672d266eb1c5a637de034f73979d429
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[sagemaker-bot]

AWS CodeBuild CI Report

• CodeBuild project: sagemaker-pytorch-inference-toolkit-pr
• Commit ID: 0d293f47b3860883ceb10dcf4ae72b167d9bc2b9
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[ashishgupta023]

have you tested below inference use cases with the DLC container

1) customer provides an inference script with context (new way) ? 2) customer provides an inference script without the context (old way) ?

could you please attach the test details on the description ?

[ashishgupta023]

I think this change will also be required for the MXNET DLC containers with MMS, instead of adding a new transformer and adapting the handler service in the pytorch toolkit, could we consider adapting the transformer and handler service in the inference toolkit to work with the context so that this change could be applied to both ? It would also help make it less error prone in the future since the change would be at a single place.

[sagemaker-bot]

AWS CodeBuild CI Report

• CodeBuild project: sagemaker-pytorch-inference-toolkit-pr
• Commit ID: ef19cb7537e9380ea32e0a70e3ed1258d71cdacd
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[waytrue17]

I think this change will also be required for the MXNET DLC containers with MMS, instead of adding a new transformer and adapting the handler service in the pytorch toolkit, could we consider adapting the transformer and handler service in the inference toolkit to work with the context so that this change could be applied to both ? It would also help make it less error prone in the future since the change would be at a single place.

Makes sense. I will split the code and re-run some tests. Will post the test results afterward

[sagemaker-bot]

AWS CodeBuild CI Report

• CodeBuild project: sagemaker-pytorch-inference-toolkit-pr
• Commit ID: ef19cb7537e9380ea32e0a70e3ed1258d71cdacd
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[sagemaker-bot]

AWS CodeBuild CI Report

• CodeBuild project: sagemaker-pytorch-inference-toolkit-pr
• Commit ID: ef19cb7537e9380ea32e0a70e3ed1258d71cdacd
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[sagemaker-bot]

AWS CodeBuild CI Report

• CodeBuild project: sagemaker-pytorch-inference-toolkit-pr
• Commit ID: ef19cb7537e9380ea32e0a70e3ed1258d71cdacd
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository