Process several images in one request

BraginIvan commented 2 years ago

Context

Cant send several images in a single request

first request

$ curl http://localhost:8080/predictions/resnet-18 -F 'data=@docs/images/dogs-before.jpg' -F 'data=@docs/images/kitten_small.jpg'
{
  "tabby": 0.7557169795036316,
  "tiger_cat": 0.14208266139030457,
  "Egyptian_cat": 0.059956926852464676,
  "lynx": 0.0372869074344635,
  "tiger": 0.0014757852768525481

second request

$curl http://localhost:8080/predictions/resnet-18 -F 'data=@docs/images/kitten_small.jpg' -F 'data=@docs/images/dogs-before.jpg'
{
  "toy_terrier": 0.09392645955085754,
  "bluetick": 0.07842477411031723,
  "Boston_bull": 0.05028736591339111,
  "kelpie": 0.043856214731931686,
  "German_short-haired_pointer": 0.043389711529016495
}

But I receive only one output for any request. I thought that may be there are issues with batching, but batch_size =4, delay 5000, so after request I receive the json in 5 seconds

torchserve version: master commit 6a191f822f2e759c2aad07b8f9eacbedcb039827
torch-model-archiver version: 0.5.3
torch version: 1.9.0+cu111
torchvision version [if any]: 0.10.0+cu111
torchtext version [if any]: 0.10.0
torchaudio version [if any]: 0.9.0
java version: openjdk 11.0.14 2022-01-18
Operating System and version: openjdk 11.0.14 2022-01-18

Your Environment

pull fom master and run resnet-18 example

Expected Behavior

Expect to receive predictions for several images

Current Behavior

It returns only one prediction

Possible Solution

I found a unit test, and it works https://github.com/pytorch/serve/blob/master/ts/torch_handler/unit_tests/test_image_classifier.py#L30 So I don't know the solution. May be my expectations are wrong and it is correct behavior

msaroufim commented 2 years ago

Can you try this syntax for your curl requests. Often I've found it confusing where some calls are actually synchronous

https://github.com/pytorch/serve/blob/master/test/pytest/test_handler.py#L246

BraginIvan commented 2 years ago

It will make 2 requests, right? I did many async requests with aiohttp and it works great, I just want to send several inputs in one http request. Like in the documentation.

import requests

res = requests.post("http://localhost:8080/predictions/squeezenet1_1", files=[{'data': open('docs/images/dogs-before.jpg', 'rb')}, {'data': open('docs/images/kitten_small.jpg', 'rb')}])

This also returns one result

msaroufim commented 2 years ago

It will make 2 requests, right? Yes

I just want to send several inputs in one http request

Oh I see interesting, so you're not batching 2 examples the examples are just related in some way. In your case you'd probably need to create a custom handler. I'd start with creating your own custom handler and seeing if both images are making it to the preprocess() function. A good but complex handler to follow as an example is this one https://github.com/pytorch/serve/blob/master/examples/Huggingface_Transformers/Transformer_handler_generalized.py#L110

BraginIvan commented 2 years ago

Actually I don't need it, I just found it in documentation. May be it makes sense to remove it from documentation. Are there any cases when it may simplify work? May be it will be slightly faster to make 1 connection instead of several, but you win nanoseconds.

msaroufim commented 2 years ago

Yeah I think the only real use case is supporting sequential data like time series, without that use case multiple requests into one is cool but not something that unlocks some great capability

pytorch / serve