search

aws / sagemaker-tensorflow-serving-container

A TensorFlow Serving solution for use in SageMaker. This repo is now deprecated.
Apache License 2.0
174 stars 101 forks source link

ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (502) from primary with message "<html> <head><title>502 Bad Gateway</title></head> #222

Closed jocelynbaduria closed 2 years ago

jocelynbaduria commented 2 years ago

I created two model with different endpoints for predictions. The first endpoint works fine in prediction. the second one did not.

My first input model is accepting 3channels of images and the second model only accepts one channel of images

When I run the first prediction using 3channels of images It is doing a prediction. The second prediction using the second endpoint got an error.

ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message "{ "error": "input and filter must have the same depth: 3 vs 1\n\t [[{{node conv2d_1/Conv2D}}]]" }

So I used different image with one channel for the second endpoint prediction but I got this new error ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (502) from primary with message " 502 Bad Gateway

Picture1

Expected behavior What I wanted to happen is I can used my first prediction from first endpoint using 3channel and used that predicted image to my second endpoint as one channel image.

System information A description of your system. Please provide: tensorflow==1.15.0 h5py==2.10.0 keras-preprocessing==1.1.0 numpy==1.16.6 framework_version = '1.15.2'

Custom Docker image (Y/N): N

Additional context I am using only one notebook instances for these two Tensorflow model endpoints. The first model endpoint is a dlib model converted to Tensorflow and the second model is Haar Cascade Pre-trained model Tensorflow.

jocelynbaduria commented 2 years ago

I resolve the issue.

satishpasumarthi commented 2 years ago

@jocelynbaduria Could you please share the resolution steps you followed so that it might help others who run into similar issue? Thanks

jocelynbaduria commented 2 years ago

@satishpasumarthi I was able to resolve it by making sure the inputs for endpoints of predictors are the same as the required inputs by the saved model. The inputs shape channel should be the same, example if your input shape image is [-1, 130, 130, 1] the input shape channel for your predictor is the same as this input which is 1.

Screen Shot 2022-06-25 at 1 45 15 PM Screen Shot 2022-06-25 at 1 46 08 PM