Cloudwatch does not log stack trace when error occurs during invocation.

[Olamyy]

Describe the bug When a code related error comes up during model invocation, the stack trace of the error is not logged on cloudwatch. The exception from sagemaker points to a cloudwatch url alright but the cloudwatch logs in the attached url does not show the trace of the error. My current work around is to manually catch and print the stack trace.:

try:
    from a.b import c
except Exception as error:
    print(error)
    print("First attempt failed. Trying second.")
    exc_type, exc_obj, exc_tb = sys.exc_info()
    fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
    print(exc_type, fname, exc_tb.tb_lineno)
    print(traceback.format_exc())

    from A.a.b import c

To reproduce

1. Deploy a model making sure the entry_point has some python error. (Like an import)
2. Invoke the deployed endpoint.

Expected behavior The stack trace should be logged in cloudwatch.

Screenshots or logs If applicable, add screenshots or logs to help explain your problem. When you go to the attached URL, the stack trace is not found there.

System information A description of your system. Please provide:

• SageMaker Python SDK version:1.56.2
• Framework name (eg. PyTorch) or algorithm (eg. KMeans): Pytorch
• Framework version: 1.5.0
• Python version: 3.7
• CPU or GPU: CPU
• Custom Docker image (Y/N): N

[laurenyu]

sorry for the slow response here - looks like the code that's causing the issue is https://github.com/aws/sagemaker-inference-toolkit/blob/master/src/sagemaker_inference/transformer.py#L90. It's not clear why the stacktrace was swallowed in the first place, so I think we should be able to add it back.

[laurenyu]

I've opened a PR to try and address this: https://github.com/aws/sagemaker-inference-toolkit/pull/51

[laurenyu]

PR has been merged. the change should be present during the next release of the PyTorch/MXNet inference images