NBoost is a scalable, search-api-boosting platform for deploying transformer models to improve the relevance of search results on different platforms (i.e. Elasticsearch)
Apache License 2.0
675
stars
69
forks
source link
Nboost does not handling concurrent user request #81
I have tried load testing after running nboost services using pt-tinybert-msmarco default model, but it is not handling it even at 10-20 requests at a time. Seems like it is processing all requests at sequentially.
isn't it not salable? Or if need to done some configuration changes to make it handle multiple requests at production level?
I have tried load testing after running nboost services using pt-tinybert-msmarco default model, but it is not handling it even at 10-20 requests at a time. Seems like it is processing all requests at sequentially. isn't it not salable? Or if need to done some configuration changes to make it handle multiple requests at production level?