The second response has a shape which was unbatched but is missing the data, the first response has data for both responses by only the shape for the first response.
This can be resolved by setting the shape of requests to [1] instead of [len(bytes)].
Could you check whether the number of string elements is the correct shape and if adapt the example accordingly?
I played around with the custom json example and adaptive batching and found that adaptive batching only behaves correctly if we specify the shape of the input data (one byte array/string) instead of len(bytes) which the example does (here: https://github.com/SeldonIO/MLServer/blob/ce6c0de2fde54026bb9d5fe65ab1291927512266/docs/examples/custom-json/README.ipynb?short_path=475bbb5#L203C16-L203C17 ) the adaptive batching functionality will fail and incorrectly unbatch the request.
I extended the example custom-json with adaptive batching
My modified code can be found here: https://gist.github.com/patrickhuy/3fbdf9c4f4d483826f838aac859ebbbb
My client sends 2 requests:
These are batched and my custom model component receives a request in the form of
and replies with
The client then receives 2 responses like this:
The second response has a shape which was unbatched but is missing the data, the first response has data for both responses by only the shape for the first response.
This can be resolved by setting the shape of requests to [1] instead of [len(bytes)]. Could you check whether the number of string elements is the correct shape and if adapt the example accordingly?