Closed RawthiL closed 1 month ago
Given a minimum acceptable speed of tokens per second (configured in the Manager) the Sampler should provide the Requested with a time out value that is suitable for each of the generated prompts.
Given a minimum acceptable speed of tokens per second (configured in the Manager) the Sampler should provide the Requested with a time out value that is suitable for each of the generated prompts.