Closed cheng92hao closed 3 months ago
What is the specific use case you have in mind? If you want to discuss privately, I am available in our Discord server
The points we are considering in our company's gateway research include:
- For people who need distributed load balancing, our thought is for the route feature to support such use cases
- This is in the work
Currently, I have conducted some tests and found that BricksLLM's performance is indeed much stronger than LiteLLM's. However, there are still some functional imperfections. What I would also like to understand is: if I implement load balancing at the upper layer, does BricksLLM support deploying multiple instances?
Should I rephrase the question as does BricksLLM stay consistent in a distributed environment? The answer is yes. You would need to be connected to the same DB. There might be bottlenecks with the max concurrent DB connections in a distributed environment, which you would need to handle yourself.
Does this answer your question?
Should I rephrase the question as does BricksLLM stay consistent in a distributed environment? The answer is yes. You would need to be connected to the same DB. There might be bottlenecks with the max concurrent DB connections in a distributed environment, which you would need to handle yourself.
Does this answer your question?
Yes, thank you very much! By the way, does BricksLLM support custom pricing for models, e.g. 1、cost per token ;2、cost per second.... For custom models, I would like to set the billing method by myself.
@cheng92hao N
Should I rephrase the question as does BricksLLM stay consistent in a distributed environment? The answer is yes. You would need to be connected to the same DB. There might be bottlenecks with the max concurrent DB connections in a distributed environment, which you would need to handle yourself. Does this answer your question?
Yes, thank you very much! By the way, does BricksLLM support custom pricing for models, e.g. 1、cost per token ;2、cost per second.... For custom models, I would like to set the billing method by myself.
Not yet, but it's on our roadmap. We plan to allow you to specify a custom cost for prompt tokens and completion tokens, so we can calculate costs for custom models.
By the way, if you're interested in using BricksLLM for your company, feel free to chat with us here, we can provide 1-on-1 support and implement custom features for enterprise customers :)
@cheng92hao N
Should I rephrase the question as does BricksLLM stay consistent in a distributed environment? The answer is yes. You would need to be connected to the same DB. There might be bottlenecks with the max concurrent DB connections in a distributed environment, which you would need to handle yourself. Does this answer your question?
Yes, thank you very much! By the way, does BricksLLM support custom pricing for models, e.g. 1、cost per token ;2、cost per second.... For custom models, I would like to set the billing method by myself.
Not yet, but it's on our roadmap. We plan to allow you to specify a custom cost for prompt tokens and completion tokens, so we can calculate costs for custom models.
By the way, if you're interested in using BricksLLM for your company, feel free to chat with us here, we can provide 1-on-1 support and implement custom features for enterprise customers :)
Thank you for your answer, We are currently conducting some research and testing. If there are any questions later, I would be happy to contact you. By the way, If there were more detailed official documentation, that would be perfect.
Recently, I have been studying the large model gateways LiteLLM and BricksLLM, we will use it in production, and I would like to ask some questions:
1、Why does BricksLLM not have load balancing using multiple BricksLLM instances? Will you support it in the future; 2、Why does BricksLLM not have load balancing using multiple instances of the Router object? There is only one simple fallback routing policy