search

pytorch / serve

Serve, optimize and scale PyTorch models in production
https://pytorch.org/serve/
Apache License 2.0
4.23k stars 864 forks source link

High utilization of hardware #1851

Closed Vert53 closed 2 years ago

Vert53 commented 2 years ago

HI, I'm trying to use torchserve as a backend with a custom hardware setup. How do you suggest to run such that the hardware is maximally utilized? For example I tried using the benchmarks-ab.py script to test the server for throughput on resnet18 but only achieved ~200 requests per second (tried different batch sizes) while the hardware is capable of crunching at least 10,000 images per second.

Thanks for any help.

msaroufim commented 2 years ago

Hi @Vert53 I need a bit more context to properly answer your question. What kind of hardware are you using? Which model? The template in the Github issue is a good start. If for whatever reason you can't share more details and work for a custom hardware provider would be happy to chat more via email so lmk

Vert53 commented 2 years ago

@msaroufim Chatting privately would be great as we are looking to support torchserve for our processor. Where can I contact you?

msaroufim commented 2 years ago

First Name last name at fb.com

Vert53 commented 2 years ago

@msaroufim I sent you a message on on facebook.