I've tested the project with ab and found out that the parallel capacity is very low(when sending 50 concurrency with ab it can only make about 2-3 parallel process).
And when I check the VRAM,it only occupied about 4GB.So I was wondering if you would add argument(function) to allow specify parallel capacity(more model workers)?Or you can just take as much VRAM as possible to create more model workers(like VLLM's do).
Thanks for the great job you've done and looking forward for your reply!
mercuryyy
commented
1 month ago
I've tested the project with ab and found out that the parallel capacity is very low(when sending 50 concurrency with ab it can only make about 2-3 parallel process). And when I check the VRAM,it only occupied about 4GB.So I was wondering if you would add argument(function) to allow specify parallel capacity(more model workers)?Or you can just take as much VRAM as possible to create more model workers(like VLLM's do). Thanks for the great job you've done and looking forward for your reply!