Disable caching for individual calls

[tom-doerr]

I'm trying to disable the cache for an inference call. I tried setting different temperature values but DSPy still uses the cache for those calls.

                temperature = 2.7 + (1 * random.random())
                result = self.predictor_cot(input_text=input_text, temperature=temperature)

                temperature = 2.7 + (1 * random.random())
                with dspy.settings.context(lm=model, trace=[], temperature=temperature):
                    result = self.predictor_cot(input_text=input_text)

[okhat]

@tom-doerr you need to pass temperature to the LM, not the context or the predictor. That said, I think it would be nice if one could set the LM kwargs in easier ways.

[vection]

@tom-doerr you need to pass temperature to the LM, not the context or the predictor. That said, I think it would be nice if one could set the LM kwargs in easier ways.

@okhat can you elaborate please how to do it? did you mean like this:

llm = dspy.OpenAI(model=model_type, max_tokens=self.MAX_TOKENS, temperature=temperature+0.001)
dspy.settings.configure(lm=llm)

whenever I want to clear the cache for specific response? because its not working for me.

In my opinion need solution to run-time cases as sometimes need to send same prompt to different LLMs but it still pulling the cached solution without notice the change of LLM.

[okhat]

Supported in DSPy 2.5 https://github.com/stanfordnlp/dspy/blob/main/examples/migration.ipynb