tf serving input

[marziehoghbaie]

hi, I'm using tensorflow serving for mtcnn and I want to use batch request to reduce inference time, but one of the networks(pnet) can accept inputs with different sizes. when I send that request, I get this error: { "error": "Failed to process element: 1 key: input_4 of 'instances' list. Error: Invalid argument: Expecting tensor size: 82944 but got: 41616" } it seems that served models can only handle batch requests if all the dimension of instances are the same, is it true?

[marziehoghbaie]

is it possible to send input size for each instance in the batch?

[gowthamkpr]

@marziehoghbaie Can you gives us a minimal code for us to reproduce this issue. Thanks!

[marziehoghbaie]

Hi @gowthamkpr, Yes, I'll send my code tomorrow morning

[marziehoghbaie]

I want to send all scales at once to get the responses at once: This is my for loop for creating the request: for i, scale in enumerate(scales): resized_h = int(h * scale) resized_w = int(w * scale) resized_img = cv2.resize(data, (resized_w, resized_h)) img_padd = self.add_padding(resized_img, desired_h=desired_h, desired_w=desired_w, w=resized_w, h=resized_h) img_padd = tf.expand_dims(img_padd, axis=0); inpp.append({"input_4": np.squeeze(img_padd.numpy()[0]).tolist()}) t = time() requests_data = json.dumps({"signature_name": "serving_default", "instances": inpp}); json_response = requests.post("http://" + self.host + ":" + self.ports["pnet"] + "/v1/models/pnet:predict", data=requests_data, headers=self.headers);

[nrobeR]

Are you passing instances or inputs in the JSON object? Based on the documentation https://github.com/tensorflow/serving/blob/master/tensorflow_serving/util/json_tensor.h#L66-L69, 'instances' requires all inputs to have same 0-th dimension while 'inputs' can represent input tensors with varying sizes.

[nrobeR]

Hi @marziehoghbaie, could you please try with inputs https://github.com/tensorflow/serving/blob/master/tensorflow_serving/util/json_tensor.h#L137-L144?

[marziehoghbaie]

Hi @nrobeR, to answer your 1st question, yes I pass the inputs as JSON objects and it works perfectly when I resize the image inputs so they have the same size, But my model must process inputs with different size. As the document says all the inputs with the same name, e.g. inp_1, must have the same size. For the second question, I can't see the point? this example has three inputs with different names, types, and size('tag', 'signal', 'sensor'); It's not related or may be I didn't get that. Could you explain your point?

[pindinagesh]

@marziehoghbaie

Could you please have a look at this issue and serving_input_function for reference and let us know if it helps? Thanks!

[pindinagesh]

@marziehoghbaie

Closing this issue due to inactivity. Please feel free to reopen if this still exist. Thanks