Uma maneira de tentar novamente, caso a API resulte em erros. Por exemplo, ao devolver um JSON com formato inválido
Um retry devido a timeout
Algumas ideias inicias já foram discutidas. A thread completa segue abaixo
p3nGu1nZz — Yesterday at 5:12 PM
having a built in retry mechanism to generate reasponses from models you pull down would be handy, alot of times i have to build that manually into my ollama calls, as i dont think ollama has api for this?
rrg92 — Yesterday at 10:22 PM
I'm not aware of any retry mechanism. What conditions do you use for retry? Is there a specific message format? When I need to use it, I also implement it manually.
But this would be relatively simple to implement. We could have a parameter that checks something in the response to decide
p3nGu1nZz — Today at 11:32 AM
Ya i didnt think so.
i typically just sniff the raw response string for starting with Error: , or JSON decode error. Then have my models always return their content back as json or xml, and i do a hard fail if i do not get json or xml back, llama and phi seem to get about 85%+ accuracy when returning structured data, so in case of error, i just try again. In python you can easily do this using the tenacity package.
i definitely know i am going to be using this type of retry logic for everf function i make on my model Tau, so i def can see this as a useful feature.
i would leave as an optional parameter with a default of 1.
rrg92 — Today at 11:35 AM
Hmm, I understand! You're referring to attempts when the structure of the returned data cannot be converted to the desired schema (or can only be converted to the desired type)... In those cases, you ask the LLM again... when I need to do this, I implement it manually, but I think PowershAI could have something that makes this easier.
p3nGu1nZz — Today at 11:37 AM
100% you got it
rrg92 — Today at 11:37 AM
I thought I could implement a parameter where you can pass an expected response. It could be a simple string with a wildcard, a regex, an exception, or even a script that allows you to create more complex logic. Then, PowerShell could attempt this a certain number of times (also configurable via a parameter). What do you think about that?
p3nGu1nZz — Today at 11:38 AM
ya or inject a Custom Error Object into the call which has this data, but thats super fancy
i forgot to mention having a timeout in seconds would be helpful too.
rrg92 — Today at 11:39 AM
maybe somehting like that ? (in complex) cases
p3nGu1nZz — Today at 11:40 AM
o thats a cool idea 🙂
rrg92 — Today at 11:40 AM
so simple as that
RetryLogic*
p3nGu1nZz — Today at 11:40 AM
i think retry might suffice, Log might conflict with Log api you might have
p3nGu1nZz — Today at 11:41 AM
your preference, i like shortest names possible, java folks like long name,
i cant take credit, i got the original idea from the node package SuperAgent
super handy when your load balancer forgets your responses and times out
rrg92 — Today at 11:43 AM
Nice. Timeout is a common parameter in many solution. That type of retry logic is somehting I see in many libs and apis
O @p3nGu1nZz sugeriu 2 features interessantes:
Algumas ideias inicias já foram discutidas. A thread completa segue abaixo
p3nGu1nZz — Yesterday at 5:12 PM having a built in retry mechanism to generate reasponses from models you pull down would be handy, alot of times i have to build that manually into my ollama calls, as i dont think ollama has api for this?
rrg92 — Yesterday at 10:22 PM I'm not aware of any retry mechanism. What conditions do you use for retry? Is there a specific message format? When I need to use it, I also implement it manually. But this would be relatively simple to implement. We could have a parameter that checks something in the response to decide
p3nGu1nZz — Today at 11:32 AM Ya i didnt think so. i typically just sniff the raw response string for starting with Error: , or JSON decode error. Then have my models always return their content back as json or xml, and i do a hard fail if i do not get json or xml back, llama and phi seem to get about 85%+ accuracy when returning structured data, so in case of error, i just try again. In python you can easily do this using the tenacity package.
i definitely know i am going to be using this type of retry logic for everf function i make on my model Tau, so i def can see this as a useful feature.
i would leave as an optional parameter with a default of 1.
rrg92 — Today at 11:35 AM Hmm, I understand! You're referring to attempts when the structure of the returned data cannot be converted to the desired schema (or can only be converted to the desired type)... In those cases, you ask the LLM again... when I need to do this, I implement it manually, but I think PowershAI could have something that makes this easier.
p3nGu1nZz — Today at 11:37 AM 100% you got it
rrg92 — Today at 11:37 AM I thought I could implement a parameter where you can pass an expected response. It could be a simple string with a wildcard, a regex, an exception, or even a script that allows you to create more complex logic. Then, PowerShell could attempt this a certain number of times (also configurable via a parameter). What do you think about that?
p3nGu1nZz — Today at 11:38 AM ya or inject a Custom Error Object into the call which has this data, but thats super fancy i forgot to mention having a timeout in seconds would be helpful too.
rrg92 — Today at 11:39 AM maybe somehting like that ? (in complex) cases
p3nGu1nZz — Today at 11:40 AM o thats a cool idea 🙂
rrg92 — Today at 11:40 AM so simple as that
RetryLogic* p3nGu1nZz — Today at 11:40 AM i think retry might suffice, Log might conflict with Log api you might have
p3nGu1nZz — Today at 11:41 AM your preference, i like shortest names possible, java folks like long name, i cant take credit, i got the original idea from the node package SuperAgent super handy when your load balancer forgets your responses and times out
rrg92 — Today at 11:43 AM Nice. Timeout is a common parameter in many solution. That type of retry logic is somehting I see in many libs and apis