Define Model API

[abrichr]

Feature request

We want to define a prompt-based API for interacting with language models.

This task involves defining the high level APIs that connect ReplayStrategies to Large Language Models.

Motivation

Related: https://github.com/MLDSAI/OpenAdapt/issues/242

Other

No response

[FFFiend]

For the current Replay Strats using LLM's, don't the get_completion methods serve as a kind of API to the model anyways? So perhaps we unify it into one class, with a dict of available models and paths, prompt and max input size?

Something like (sorry for the horrible indenting I wrote this in the editor in Markdown :D)

MODEL_DICT = {'LLama': {'Quantized': quantized_path, 'Normal': normal_path},...}

class GeneralReplayStrategy(BaseReplayStrategy):

          def  __init__(
                                self,
                                recording: Recording, 
                                max_output_size: int, 
                                prompt: str, 
                                max_input_size: int = len(prompt),
                                use_quantized: bool,
                                model_type: str
                ):
                    super.__init__()
                    self.recording= recording
                    self.max_input_size = max_input_size
                    self.max_token_size = max_output_size
                    self.use_quantized = use_quantized
                    self.model_type = model_type

         def get_completion():
                    if self.use_quantized:
                        # use GGML models
                        if self.model_type in MODEL_DICT:
                        ...
                    else:
                        # use regular models hosted on one of OpenAdapt's HF spaces.          

[abrichr]

See also: https://docs.google.com/document/d/15kEsWSvT3TPcudONFQnNo1f6nhArBq6Bg1k-X6ZtnKk/edit#heading=h.wlbqvhj3qiu