-
It's sad (and slow) when parallelisation is off but a generator can support it.
Proposal:
* benchmarks show `parallel_attempts` has more impact than `parallel_requests` (over a whole default run) …
-
I started a server with the command ` OLLAMA_NUM_PARALLEL=4 OLLAMA_MAX_LOADED_MODELS=4 ./ollama serve`. We open 4 terminals and executed the command` ./ollama run codellama after which the model loade…
-
Hi team,
I've been stress-testing Sarathi and pipeline parallel doesn't pass the assert statement https://github.com/microsoft/sarathi-serve/blob/f17201a8088d5819bd4398719ed51a09bd9065dd/sarathi/co…
yunoJ updated
2 weeks ago
-
As discussed in the EDV WG call, we would like to create and update several documents to a backing EDV from a Web Wallet while minimizing user visible latency (and server overhead). In general terms, …
-
if we are sending multiple requests at a time to tinyproxy, then the memory allocated for the each request is not getting released even after getting the response for the request.
and tinyproxy is no…
-
Exokit hasn't supported (or really needed to support) parallelized dom load so far, but we've been seeing some timeout issues when resources like images and scripts load too slowly (serially) on the i…
avaer updated
6 years ago
-
### Do you need to file an issue?
- [ ] I have searched the existing issues and this bug is not already filed.
- [ ] My model is hosted on OpenAI or Azure. If not, please look at the "model providers…
-
Thanks for a great library. It does make analysis to visualization much easier. When using python language kernel(with subprocess), it waits for finishing the request for one client before it starts t…
-
From my experience parallel requests are not supported in the way one would expect. **Example:** I trigger 3 requests more or less concurrently. Say, each request takes 10 seconds to complete. Now the…
-
### Do you need to file an issue?
- [x] I have searched the existing issues and this bug is not already filed.
- [x] My model is hosted on OpenAI or Azure. If not, please look at the "model provid…