search

PygmalionAI / aphrodite-engine

PygmalionAI's large-scale inference engine
https://pygmalion.chat
GNU Affero General Public License v3.0
779 stars 87 forks source link

[Usage]: How to use prefix caching? #520

Open thatname opened 2 weeks ago

thatname commented 2 weeks ago

Your current environment

I can not find any doc about this feature.

How would you like to use Aphrodite?

No response

sgsdxzy commented 1 week ago

It's called context-shift, the name might be a bit misleading because it's not the same context shift in koboldcpp, but prefix caching.

GodZioP commented 4 days ago

It's called context-shift, the name might be a bit misleading because it's not the same context shift in koboldcpp, but prefix caching.

So if I add --context-shift as an additional argument it should work, right?

sgsdxzy commented 3 days ago

Yes.