Currently get_response supports converters and validators as ways to coerce model response and retry it it's bad. But we want to support more complicated behaviors such as:
increasing the temperature on failure
switching to a bigger model on failure
showing the model it's output so it can understand why it failed and self correct
To support these let's make get_response accept a retry_strategy argument which is of abstract class RetryStrategy which should have one function decide (name TBD, feel free to choose the name of you have a good idea) which accepts the call_args the number of the previous attempt (0 indexed) and the model output as text. It then returns a tuple, (behavior, next_call_args, result) where behavior is a new enum Behavior which can be either RETRY or RETURN. If it is set to Return the text and result are added to the spice response and returned. Otherwise we try again with next_call_args. It's up to the strategy to throw an exception or otherwise define a failure case after a certain number of iterations.
This will deprecate the current converter/validator/retries arguments. We should support them for now and if passed create a RetryStrategy called Default which reproduces the current behavior.
Currently
get_response
supportsconverters
andvalidators
as ways to coerce model response and retry it it's bad. But we want to support more complicated behaviors such as:To support these let's make
get_response
accept a retry_strategy argument which is of abstract classRetryStrategy
which should have one functiondecide
(name TBD, feel free to choose the name of you have a good idea) which accepts thecall_args
the number of the previous attempt (0 indexed) and the model output as text. It then returns a tuple,(behavior, next_call_args, result)
where behavior is a new enum Behavior which can be either RETRY or RETURN. If it is set to Return the text and result are added to the spice response and returned. Otherwise we try again with next_call_args. It's up to the strategy to throw an exception or otherwise define a failure case after a certain number of iterations.This will deprecate the current converter/validator/retries arguments. We should support them for now and if passed create a RetryStrategy called Default which reproduces the current behavior.