search

microsoft / DeepSpeed-MII

MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.
Apache License 2.0
1.76k stars 164 forks source link

Performance of RESTful API #324

Open kihanc opened 7 months ago

kihanc commented 7 months ago

I have measured the performance of Mistral 7B w/ and w/o Restful API, and got 7.1 reqs/s and 0.48 reqs/s individually. (avg. input/output tokens: 232/208, A100 80GB)

mrwyattii commented 7 months ago

Hi @kihanc we are aware of performance issues related to the restful API and will be providing an update on this feature soon. Thanks

mrwyattii commented 7 months ago

@kihanc please try with the latest main branch. I have made improvements that allow us to match performance the RESTful API to our Python API (see #328).

BaiMoHan commented 7 months ago

@kihanc please try with the latest main branch. I have made improvements that allow us to match performance the RESTful API to our Python API (see #328).

Hey, how can I make Restful start and listen on 0.0.0.0 instead of localhost? I often encounter situations where I deploy my application on a remote machine but cannot test it using external API tools like Postman because it only listens on localhost.

PawanOsman commented 7 months ago

@kihanc please try with the latest main branch. I have made improvements that allow us to match performance the RESTful API to our Python API (see #328).

Hey, how can I make Restful start and listen on 0.0.0.0 instead of localhost? I often encounter situations where I deploy my application on a remote machine but cannot test it using external API tools like Postman because it only listens on localhost.

I think it's using localhost by default and there is no option to set it, you can modify this file locally until they add an option to set the hostname

BaiMoHan commented 7 months ago

@kihanc please try with the latest main branch. I have made improvements that allow us to match performance the RESTful API to our Python API (see #328).

Hey, how can I make Restful start and listen on 0.0.0.0 instead of localhost? I often encounter situations where I deploy my application on a remote machine but cannot test it using external API tools like Postman because it only listens on localhost.

I think it's using localhost by default and there is no option to set it, you can modify this file locally until they add an option to set the hostname

Thank you!

kihanc commented 6 months ago

@mrwyattii Thanks for the quick update! Your test code works perfectly on our end. But, if we hit more than 100 requests at once, things get wonky and connections are reset. It would be awesome if you could add support for concurrent connections on your API server.

mrwyattii commented 6 months ago

@kihanc if you increase the number of restful_processes (which defaults to 32) does that help? Perhaps try 64?

mii.serve(model_name, restful_processes=64, enable_restful_api=True)
kihanc commented 6 months ago

@mrwyattii I already tried that before but that was not helpful.

IKACE commented 2 months ago

Do we have any updates on this issue? I am also observing "Connection reset" failures when request rate is high