aws / amazon-sagemaker-examples

Example 📓 Jupyter notebooks that demonstrate how to build, train, and deploy machine learning models using 🧠 Amazon SageMaker.
https://sagemaker-examples.readthedocs.io
Apache License 2.0
10.14k stars 6.78k forks source link

Image-classification-fulltraining-highlevel.ipynb failed CI #2266

Open hongshanli23 opened 3 years ago

hongshanli23 commented 3 years ago

Link to the notebook: https://github.com/aws/amazon-sagemaker-examples/blob/master/aws_sagemaker_studio/sagemaker_algorithms/imageclassification_caltech/Image-classification-fulltraining-highlevel.ipynb

Error:


Exception encountered at "In [12]":

AttributeError Traceback (most recent call last)

in 6 payload = bytearray(payload) 7 ----> 8 ic_classifier.content_type = 'application/x-image' 9 result = json.loads(ic_classifier.predict(payload)) 10 # the result will output the probabilities for all classes AttributeError: can't set attribute
jkroll-aws commented 3 years ago

This serializer use is deprecated in SageMaker V2. I tried the following fix:

from sagemaker.serializers import IdentitySerializer
ic_classifier.serializer = IdentitySerializer('image/jpeg')

However, I am encountering the error:

---------------------------------------------------------------------------
ModelError                                Traceback (most recent call last)
<ipython-input-36-896cd20b2c67> in <module>
     13 
     14 #ic_classifier.serializer = IdentitySerializer()#'application/x-recordio-protobuf')
---> 15 result = json.loads(ic_classifier.predict(payload))
     16 # the result will output the probabilities for all classes
     17 # find the class with maximum probability and print the class index

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/predictor.py in predict(self, data, initial_args, target_model, target_variant, inference_id)
    134             data, initial_args, target_model, target_variant, inference_id
    135         )
--> 136         response = self.sagemaker_session.sagemaker_runtime_client.invoke_endpoint(**request_args)
    137         return self._handle_response(response)
    138 

~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
    355                     "%s() only accepts keyword arguments." % py_operation_name)
    356             # The "self" in this scope is referring to the BaseClient.
--> 357             return self._make_api_call(operation_name, kwargs)
    358 
    359         _api_call.__name__ = str(py_operation_name)

~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
    674             error_code = parsed_response.get("Error", {}).get("Code")
    675             error_class = self.exceptions.from_code(error_code)
--> 676             raise error_class(parsed_response, operation_name)
    677         else:
    678             return parsed_response

ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (503) from model with message "{
  "code": 503,
  "type": "InternalServerException",
  "message": "Prediction failed"
}
"

I found this doc but it did not solve the problem: https://docs.aws.amazon.com/sagemaker/latest/dg/neo-troubleshooting-inference.html