msaroufim
commented
1 year ago Two important parameters to tune
"responseTimeout": 1200this will be a 1,200s timeout - would suggest drastically reducing this and see if anything changes"maxBatchDelay": 100,this will wait up to 100s for your batch to be filled out, it might be the case that sending the images over the wire is taking too long which is why you're consistently seeing results of 100s. Try reducing this or whatever elements are present for the batch to be filled out will be processed
Git-TengSun
lxning
question
I deployed a model using torchserve, batchsize=8, workers=1. When eight small images(wxh:721x222) are tested at the same time, torchserve can respond quickly and process them in the same batch. When eight large images(wxh:1369x1824) are tested, torchserve will delay responding for a long time(more than 100 seconds) , although it will be processed in the same batch eventually. I think it's abnormal. I don't know what caused it.
config.properties
enable_envvars_config=true inference_address=http://127.0.0.1:8080 management_address=http://127.0.0.1:8081 metrics_address=http://127.0.0.1:8082 grpc_inference_port=7070 grpc_management_port=7071 number_of_netty_threads=32
netty_client_threads=16
job_queue_size=1000 model_store=./model-store max_response_size=2047093252 max_request_size=2047093252
number_of_gpu=4
enable_metrics_api=false
prefer_direct_buffer=true
async_logging=true
decode_input_request=false
models={ "psenet": {\ "1.0": {\ "defaultVersion": true,\ "marName": "psenet.mar",\ "minWorkers": 1,\ "maxWorkers": 1,\ "batchSize": 8,\ "maxBatchDelay": 100,\ "responseTimeout": 1200\ }\ } }
Versions
TorchServe Version is 0.6.0,installed via conda. Environment headers Torchserve branch:
torchserve==0.6.0 torch-model-archiver==0.6.0
Python version: 3.8 (64-bit runtime) Python executable: /zhangfei401/346/envs/torchserve/bin/python
Versions of relevant python libraries: captum==0.5.0 future==0.18.2 numpy==1.20.3 nvgpu==0.9.0 psutil==5.9.2 pycocotools-fix-numpy==1.0.0 requests==2.28.1 torch==1.8.1+cu101 torch-model-archiver==0.6.0 torch-workflow-archiver==0.2.4 torchaudio==0.8.1 torchserve==0.6.0 torchvision==0.9.1+cu101 wheel==0.37.1 torch==1.8.1+cu101 **Warning: torchtext not present .. torchvision==0.9.1+cu101 torchaudio==0.8.1
Java Version: openjdk version "11.0.17" 2022-10-18 OpenJDK Runtime Environment (build 11.0.17+8-post-Ubuntu-1ubuntu218.04) OpenJDK 64-Bit Server VM (build 11.0.17+8-post-Ubuntu-1ubuntu218.04, mixed mode, sharing)
OS: Ubuntu 18.04.5 LTS GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 Clang version: N/A CMake version: version 3.10.2
Is CUDA available: Yes CUDA runtime version: 11.0.221 GPU models and configuration: GPU 0: Tesla V100-SXM2-32GB GPU 1: Tesla V100-SXM2-32GB GPU 2: Tesla V100-SXM2-32GB GPU 3: Tesla V100-SXM2-32GB Nvidia driver version: 450.102.04 cuDNN version: Probably one of the following: /usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.4 /usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.4 /usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.4 /usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.4 /usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.4 /usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.4 /usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.4