Add new unit and integration tests

[sachanub]

Issue #, if available:

Description of changes:

The objective of this PR is to add new unit and integration tests. Specifically, we include the following:

• Unit test for ts_environment.py: test/unit/test_ts_environment.py
  • Added unit test to create a mock TorchServe environment and check the values of the environment variables.
• SageMaker integration tests for MNIST inductor: test/integration/sagemaker/test_mnist_inductor.py
  • test/resources/mnist/model_inductor:
    • Included model files from DLC repo i.e. code/mnist.py and torch_model.pth
  • test/integration/__init__.py:
    • Added logic to define variables for MNIST inductor script and MNIST inductor model artifact.
  • test/integration/sagemaker/test_mnist_inductor.py:
    • Added CPU test to create and test endpoint with ml.c5.9xlarge instance type.
    • Added GPU test to create and test endpoints with ml.p3.2xlarge, ml.g4dn.4xlarge and ml.g5.4xlarge instance types.
• SageMaker integration tests for multi-model endpoint (MME): test/integration/sagemaker/test_multi_model_endpoint_sagemaker.py
  • test/resources/mme:
    • Copied ResNet-18 model artifact from test/resources/resnet18/default_model.
    • Copied traced ResNet-18 model artifact from test/resources/resnet18/default_traced_resnet.
  • test/integration/__init__.py:
    • Add logic to create model artifacts for the MME tests.
  • test/integration/sagemaker/test_multi_model_endpoint_sagemaker.py:
    • Added test to check only ResNet-18 model is available on the multi-model endpoint.
    • Added test to check only traced ResNet-18 model is available on the multi-model endpoint.
    • Added test to check both models are available on the multi-model endpoint.
    • Added test to no models are available on the multi-model endpoint.
    • Added test to invoke both the ResNet-18 and traced ResNet-18 models and verify output length.
• Checks for verifying that dependencies from requirements.txt are installed:
  • requirements.txt file: This file has the transformers dependency.
    • test/resources/mnist/model_cpu/code/requirements.txt.
  • Added logic to try and import transformers in the following script:
    • test/resources/mnist/model_cpu/code/mnist.py.
  • test/integration/__init__.py:
    • Added logic to include requirements.txt in MNIST tar file.
  • test/utils/file_utils.py:

    • Added logic to include requirements.txt in tar file creation.

UPDATE:

• Refactored test/integration/__init__.py to obtain model information from test/integration/all_models_info.json and iteratively create attributes for the model directory, model script name and model tar file.

Added the following local integration tests:

• Local integration test for multi-model endpoint:
  • test/integration/sagemaker/test_multi_model_endpoint_local.py:
    • Added test to send a successful ping.
    • Added test to check that list of models is empty.
    • Added test to check that the ResNet-18 and traced ResNet-18 models are successfully loaded.
    • Added test to check that the ResNet-18 model is successfully unloaded.
    • Added test to assert failure on trying to unload a model which has already been unloaded i.e. ResNet-18.
    • Added test to assert failure on trying to load a model which has already been loaded i.e. traced ResNet-18.
    • Added test to invoke both the ResNet-18 and traced ResNet-18 models and verify output length.
• Local integration test to check that all GPU IDs are returned on multi-GPU host:
  • test/integration/__init__.py:
    • Define directory path for model_gpu_context.
  • test/resources/model_gpu_context/code/inference.py:
    • Create custom model_fn, input_fn, predict_fn and output_fn to dynamically get all the GPU IDs from the context object and their corresponding PIDs and thread IDs and write them in different CSV files. Return dummy values from the functions.
  • test/integration/local/test_model_fn_context.py:
    • Added test to check all GPU IDs are returned by the context object in the model_fn, input_fn, predict_fn and output_fn functions by reading from the CSV files created in test/resources/model_gpu_context/code/inference.py.
    • Added test to check that for a given device ID, PID and thread ID are same for the model_fn, input_fn, predict_fn, output_fn functions.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[chen3933]

I notice we don't have local integration test. Is there any use case we want to cover in local integration test?

[chen3933]

I didn't see user script overwrites input_fn or output_fn. Do we want to add those tests?

[sachanub]

I notice we don't have local integration test. Is there any use case we want to cover in local integration test?

Added the following local integration tests:

• MME tests.
• Test to check all GPU device IDs are dynamically returned by the context object in model_fn.

[sachanub]

I didn't see user script overwrites input_fn or output_fn. Do we want to add those tests?

We define custom input_fn and output_fn for this test: https://github.com/aws/sagemaker-pytorch-inference-toolkit/blob/master/test/integration/local/test_serving.py#L87. This script defines custom input_fn and output_fn functions: https://github.com/aws/sagemaker-pytorch-inference-toolkit/blob/master/test/resources/mnist/model_cpu/code/call_model_fn_once.py