Plugin SDK: buffer with request body is released before plugin can access it

🐛 Describe the bug

This is a continuation of #2498

When using Request abstraction from serving SDK, attempt to call getInputStream() will throw an exception.

The error indicates that the request body buffer is released before it is passed to the Request abstraction. Unfortunately, I haven't been able to pinpoint where exactly this happens - any help with it would be much appreciated.

Error logs

2023-09-15 2023-09-15 2023-09-15 2023-09-15 2023-09-15 20:08:34 2023-09-15 20:08:34 2023-09-15 20:08:34 2023-09-15 20:08:34 2023-09-15 20:08:34 2023-09-15 20:08:34 2023-09-15 20:08:34 2023-09-15 20:08:34 2023-09-15 20:08:34 2023-09-15 20:08:34 2023-09-15 20:08:34 2023-09-15 20:08:34 2023-09-15 20:08:34 2023-09-15 20:08:34 2023-09-15 20:08:34 2023-09-15 20:08:34 20:08:34 2023-09-15T19:08:34,596 [INFO ] pool-2-thread-5 ACCESS_LOG - /172.17.0.1:35634 "POST /actuator/loggers/missing_logger HTTP/1.1" 500 33 20:08:34 2023-09-15T19:08:34,600 [INFO ] pool-2-thread-5 TS_METRICS - Requests5XX.Count:1.0|#Level:Host|#hostname:9e777ba9c5ea,timestamp:1694804914 20:08:34 2023-09-15T19:08:34,601 [ERROR] pool-2-thread-5 org.pytorch.serve.http.HttpRequestHandler - Unknown exception 20:08:34 io.netty.util.IllegalReferenceCountException: refCnt: 0 at io.netty.buffer.AbstractByteBuf.ensureAccessible(AbstractByteBuf.java:1490) ~[model-server.jar:?] at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1419) ~[model-server.jar:?] at io.netty.buffer.AbstractByteBuf.checkDstIndex(AbstractByteBuf.java:1445) ~[model-server.jar:?] at io.netty.buffer.CompositeByteBuf.getBytes(CompositeByteBuf.java:1051) ~[model-server.jar:?] at io.netty.buffer.CompositeByteBuf.getBytes(CompositeByteBuf.java:2013) ~[model-server.jar:?] at io.netty.buffer.CompositeByteBuf.getBytes(CompositeByteBuf.java:49) ~[model-server.jar:?] at org.pytorch.serve.util.NettyUtils.getBytes(NettyUtils.java:221) ~[model-server.jar:?] at org.pytorch.serve.servingsdk.impl.ModelServerRequest.getInputStream(ModelServerRequest.java:49) ~[model-server.jar:?] at org.pytorch.serve.servingsdk.impl.ModelServerRequest.getInputStream(ModelServerRequest.java:13) ~[model-server.jar:?] at org.pytorch.serve.plugins.endpoint.actuator.loggers.LoggersEndpoint.doPost(LoggersEndpoint.java:46) ~[actuator.jar:0.0.1] at org.pytorch.serve.plugins.endpoint.actuator.ActuatorRestEndpoint.doPost(ActuatorRestEndpoint.java:56) ~[actuator.jar:0.0.1] at org.pytorch.serve.http.HttpRequestHandlerChain.run(HttpRequestHandlerChain.java:78) ~[model-server.jar:?] at org.pytorch.serve.http.HttpRequestHandlerChain.lambda$handleCustomEndpoint$0(HttpRequestHandlerChain.java:101) ~[model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?] at java.lang.Thread.run(Thread.java:833) [?:?]

Installation instructions

Did you install torchserve from source? No

Are you using Docker? Yes, pytorch/torchserve:latest

Model Packaing

Works with any model

config.properties

inference_address=http://0.0.0.0:8080 management_address=http://0.0.0.0:8081 metrics_address=http://0.0.0.0:8082 load_models=all model_store=/home/model-server/model-store plugins_path=/home/model-server/plugins default_workers_per_model=4

Versions

Environment headers

Torchserve branch:

torchserve==0.8.1b20230730 torch-model-archiver==0.8.1b20230730

Python version: 3.10 (64-bit runtime) Python executable: /Users/rodionmartynov/Documents/oss/serve/venv/bin/python

Versions of relevant python libraries: captum==0.6.0 numpy==1.24.3 psutil==5.9.5 pygit2==1.12.1 pylint==2.6.0 pytest==7.3.1 pytest-cov==4.1.0 pytest-mock==3.10.0 requests==2.31.0 requests-toolbelt==1.0.0 torch==2.0.1 torch-model-archiver==0.8.1b20230730 torch-workflow-archiver==0.2.9b20230730 torchaudio==2.0.2 torchdata==0.6.1 torchpippy==0.1.1 torchserve==0.8.1b20230730 torchtext==0.15.2 torchvision==0.15.2 transformers==4.30.0 wheel==0.41.0 torch==2.0.1 torchtext==0.15.2 torchvision==0.15.2 torchaudio==2.0.2

Java Version:

OS: Mac OSX 13.4.1 (arm64) GCC version: N/A Clang version: 14.0.3 (clang-1403.0.22.14.1) CMake version: version 3.27.4

Versions of npm installed packages: markdown-link-check@3.11.2 UNMET PEER DEPENDENCY newman@5.3.2 newman-reporter-html@1.0.5

Repro instructions

Create a plugin with a model endpoint

public class LoggersEndpoint extends ModelServerEndpoint {
@Override
public void doPost(Request req, Response res, Context ctx)
  throws ModelServerEndpointException, IOException {
req.getInputStream();
}
}

Compile it into a jar and add path to jar to config.properties
Run the server (I am using docker)
Send a POST request with any body to the plugin endpoint

Possible Solution

No response

pytorch / serve