Closed ontheway16 closed 2 years ago
According to your log everything is working fine. Please provide more details.
I never had to touch multiple instances, one should be sufficient. Preferred batchsize should be actual "batched" sizes. Try [4,16].
It is also very hard to CPU bottleneck tritonserver. It is probably your client script which is bottlenecking.
Please provide output of perf_client tests, look it up in the docu.
According to your log everything is working fine. Please provide more details.
Hi Philipp,
The log clearly indicates there are 2 instances, despite the 8 setting in config. That was my point. If log says 2 instances, isnt it mean there are 2 instances for certain?
�[35mtritonserver_1 |�[0m I1012 18:15:53.739459 1 plan_backend.cc:649] Created instance yolov4_0_0_gpu0 on GPU 0 with stream priority 0
...
...
...
�[35mtritonserver_1 |�[0m I1012 18:15:53.742868 1 plan_backend.cc:649] Created instance yolov4_0_1_gpu0 on GPU 0 with stream priority 0
It is also very hard to CPU bottleneck tritonserver. It is probably your client script which is bottlenecking.
I think the NMS code in the client (processing.py) is causing this CPU bottleneck. any hopes for GPU based NMS ?
Your triton log says:
�[35mtritonserver_1 |�[0m instance_group {
�[35mtritonserver_1 |�[0m count: 2
�[35mtritonserver_1 |�[0m kind: KIND_GPU
�[35mtritonserver_1 |�[0m }
Check that you're loading the right config
It appears you are right, celery dockerized one was using a copy of model directory, somewhere else... Sorry for this inconvenience..
Tried dynamic batching since no performance gains even with 12 instances, but it says;
[35mtritonserver_1 |[0m E1012 21:33:08.045394 1 model_repository_manager.cc:1633] unable to autofill for 'yolov4', configuration specified max-batch 4 but TensorRT engine only supports max-batch 1
Is this coming from a setting in main.cpp ? Or yolov4 model cannot/should not be batched ?
Set this to at least 16 and make sure you only have one instance, or you will run out of VRAM. Triton suggests you use dynamic batching over more instances in the docu.
[4,8,16] can work better. You can test performance very accurately with the perf_client. Would be appreciated if you could do that and report the speedup that you get so others can see the benefit of dynamic batching. Tutorial is in the README of this repo.
Thanks for indicating this, first I will switch back from custom http client to original grpc one, and see if there are performance gains. I am still thinking, topping the CPUs probably be coming from NMS code..? What do you think?
High CPU load is due to preprocessing I think. It resizes your images, so this already will depend on the resolution of the images that you use (Resizing FullHD can take a CPU a few milliseconds). If you want to lower CPU usage you probably need to optimize the preprocessing method.
You can try to benchmark this particular method to see how much of it your CPU can take and then start to optimize. And check postprocessing (NMS etc.) as well, but I don't think that's the bottleneck.
after some tests, it appears, celery operations are creating the visible cpu load, tried with a normal (non-celery) triton server, with multiple files in multi loop, all processes (pre, normal and post) consumed about 10%, triton itself was like 1.4%. Under celery, this was around 25-35. Also saw there is no difference between http and grpc.
Reopen if you still have any issues with this
Hi,
After dealing long with kubernetes based solutions, I have switched to celery. currently I am able to feed the tritonserver (20.08) )good enough, despite some CPU bottleneck. But GPU is still staying on low side, utiilization ranging between 20-60%.
I was already setting config to 8 instances, but decided to give dynamic_batching a try and set the config to following;
It failed to function and I take a look to triton logs, found this;
Apparently, my 8 instances is not in place, either. Tritonserver allocates only 2.6GB GPU memory, out of 11.
Is there anything I can do, to fix model instances and dynamic_batching ? I believe if I can run more than two instances, I will not need dyn_batch stuff..
edit. quoted config values above were wrong, fixed.