📚 Documentation: Are custom JSON bodies possible with custom handlers? e.g. multiple instances per request, multi-modality instances, etc.

[j0rd1smit]

📚 Documentation: Are custom JSON bodies possible with custom handlers? e.g., multiple instances per request, multi-modality instances, etc.

I have been reading the documentation, but I'm still left with the following questions:

1) How do I format a REST API request if the model expects multiple inputs? For example, the model expects two or more images/tensors as input. Should this be done by uploading multiple files in the request?

import requests

res = requests.post(
    "http://127.0.0.1:8080/v1/models/resnet-18:predict",
    files={"image1": open("kitten.jpg", "rb"), "image2": open("kitten2.jpg", "rb")},
)

2) Is it possible to send multiple instances in single rest requests? In TensorFlow-Serving, it is possible to send multiple instances in the same request as if the JSON body has the format:

{
    "instances": [
        {"input_name": "1"},
        {"input_name": "2"},
    ]
}

Is there something similar possible in PyTorch-Serve?

3) Is it only possible to send input instances via the REST API as files? In all the documentation and examples I have seen, all the instances are sent via file uploads using either curl URL -T file_path or requests.post(URL, files={...}). I was wondering if it is possible to use a custom handler to accept, for example, a JSON body in the request. This would make it possible to solve questions 1 and 2 using a custom handler.

[HamidShojanazeri]

cc: @lxning @msaroufim

[lxning]

@j0rd1smit

1. How do I format a REST API request if the model expects multiple inputs? Example: requests.post("http://localhost:8080/predictions/squeezenet1_1", files={'data': open('docs/images/dogs-before.jpg', 'rb'), 'data': open('docs/images/kitten_small.jpg', 'rb')})

2. Is it possible to send multiple instances in single rest requests? No.

3. Is it possible to use a custom handler to accept, for example, a JSON body in the request? Yes. Torchserve envelope support it.

[lxning]

@j0rd1smit Please feel free to reopen this ticket if any further help is needed.