Open ss2342 opened 4 months ago
@ss2342 can you try adding the following to init and check?
self.kwargs = {
"temperature": 0.0,
"max_tokens": 150,
"top_p": 1,
"frequency_penalty": 0,
"presence_penalty": 0,
"n": 1,
**kwargs,
}
or make it like this:
from dsp import LM
from huggingface_hub import InferenceClient
class CustomLMClient(LM):
def __init__(self, model_endpoint, **kwargs):
self.provider = "default"
self.client = InferenceClient(model=model_endpoint)
self.kwargs = {
"temperature": 0.0,
"max_tokens": 150,
"top_p": 1,
"frequency_penalty": 0,
"presence_penalty": 0,
"n": 1,
**kwargs,
}
self.history = []
def basic_request(self, prompt, **kwargs):
kwargs = {**self.kwargs, **kwargs}
output_text = self.client.text_generation(prompt=prompt, **kwargs)
self.history.append({
"prompt": prompt,
"response": output_text,
"kwargs": kwargs
})
return output_text
def __call__(self, prompt, only_completed=True, return_sorted=False, **kwargs):
response = self.basic_request(prompt, **kwargs)
completions = [response]
return completions
and I know you mentioned when you added it in init it gave error, can you provide the error traceback for it?
another solution could be adding super().__init__(model=model)
in your __init__
since LM
base class already has a default kwargs
inside.
https://github.com/stanfordnlp/dspy/blob/4c6a2ffb6e9dd64dd6e5a587b468c695487f9136/dsp/modules/lm.py#L8
I had the same issue would be great to update the documentations here: https://dspy-docs.vercel.app/docs/deep-dive/language_model_clients/custom-lm-client
I am trying to build a custom Local Model Client as per the following documentation . This is how I how I have set it up:
I am not using the built-in
dspy.HFClientTGI
as it requires a model argument which is the model name on HuggingFace. However, this model is not on the huggingface hub and is already deployed on a TGI endpoint. However, I encounter the following error when trying to use it withdspy.ChainOfThought
.This is the code I tried:
This is my traceback:
I am a bit confused since it doesn't mention in the docs to pass in
kwargs
to the__init__
function. I did try also doing that but that did not work either.I am using
dspy-ai==2.4.0
In addition to this, I have a model in which I have added in custom tags (similar to the [INST][/INST] for mistral instruct). How do I ensure that the optimization prompt takes these special tags into account?