Open RamVegiraju opened 5 months ago
Following example: https://github.com/deepjavalibrary/djl-demo/blob/2a5152f578f5954b8b68acdee18eed4e2a75c81f/aws/sagemaker/large-model-inference/sample-llm/trtllm_rollingbatch_deploy_llama_13b.ipynb, errors out with a health check error with current code, the following cell needs to be uncommented for successful deployment:
model.deploy(initial_instance_count=1, instance_type=instance_type, endpoint_name=endpoint_name, container_startup_health_check_timeout=3600 ##uncomment )
Following example: https://github.com/deepjavalibrary/djl-demo/blob/2a5152f578f5954b8b68acdee18eed4e2a75c81f/aws/sagemaker/large-model-inference/sample-llm/trtllm_rollingbatch_deploy_llama_13b.ipynb, errors out with a health check error with current code, the following cell needs to be uncommented for successful deployment: