Raising HTTPexception on single request from encode_request or decode_request raise same for all request in batch

[bimarshak7]

🐛 Bug

To Reproduce

Setup litserve api with batch processing (max_batch_size>1) and raise HTTPexception (from fastapi) either in encode_request or decode_request.

Code sample

def decode(self, output):
        if "error" in output:
            raise HTTPException(status_code=500, detail="Error processing file")

        return output

Expected behavior

Only the request with error should return 500 and other request should return output as expected.

[bimarshak7]

@Paidinful I changed my code to handle batch inputs. the pipeline includes batching and unbatching in the middle few times and in the process some input may be unprocessable which i remove from batch and continue processing. Even the output passed to decode function is correct. But if any request should be given 500, then all request will get 500 in response

[aniketmaurya]

hey @bimarshak7, LitServe currently process the whole decode - predict - encode pipeline together and failure of a single item leads to failure of all the requests in that batch. Thanks for reporting and we will let you know when we fix this.

[bimarshak7]

@aniketmaurya I am waiting for the update