search

Xilinx / inference-server

https://xilinx.github.io/inference-server/
Apache License 2.0
43 stars 13 forks source link

Batcher produces batch larger than requested size #53

Closed varunsh-xilinx closed 2 years ago

varunsh-xilinx commented 2 years ago

Describe the bug The batcher can produce a batch that's larger than the requested batch size. Reported by @bpickrel

To Reproduce Steps to reproduce the behavior:

  1. Set the batch size to 64
  2. Load a worker (use the timeout parameter to enforce a long batch timeout period)
  3. Send two requests with 62 images and 4 images
  4. Observe the batch size

Expected behavior It should result in two requests, instead of one request with 66 images

Additional context There's another open question here about what the server should do if a user request has more images than the configured batch size.