Open sshleifer opened 5 months ago
I do not believe there is currently a way to disable sliding window, but I think this is something we should add
You can disable the sliding window by using --disable-sliding-window
. For mistral, as you've done, you'll need to restrict the model to a context window of 4096 tokens to do this.
@robertgshaw2-neuralmagic considering that prefix caching is by definition focusing the early portion of the prompt whereas the sliding window in mistral only kicks in after 4096 tokens, do you think it might be possible to enable a prefix cache that only looked at the first 4096 tokens of a prompt so there wasn't a clash? That would be the best of both worlds here.
raises
NotImplementedError: Sliding window is not allowed with prefix caching enabled!
Is there a way to turn off sliding window and keep prefix caching?
(More generally is there a list of commands to serve common models efficiently?)