[Feature] Cache for LLMs - Githubissues

ai-for-java / ai4j

Apache License 2.0

26 stars 3 forks source link

[Feature] Cache for LLMs #21

Open deep-learning-dynamo opened 1 year ago

deep-learning-dynamo commented 1 year ago

As a java developer I want to be able to cache responses from LLM:

to speedup frequent prompts
to save noney

Requirements:

easy way to plug in custom cache providers
simple cache: match the prompt with simple heuristics: ignoring case / ignoring whitespaces / etc
semantic cache: by semantic similarity with configurable threshold