DNGros / lmwrapper

An object-oriented wrapper around language models (like openai endpoints or huggingface)
1 stars 1 forks source link

Add api for sampling more than once #31

Open DNGros opened 7 months ago

DNGros commented 7 months ago

Add some builtin support for generating multiple generations. This leaves room for more efficient generation on backends like HF (you only need to encode once). Also can potentially cache multiple generations (ideally in clever way that lets you grow the number of generations and reuse prior caches).