search

stanfordnlp / dspy

DSPy: The framework for programming—not prompting—foundation models
https://dspy-docs.vercel.app/
MIT License
16.46k stars 1.27k forks source link

Implement Exponential Backoff for ThrottlingException in InvokeModel Calls #768

Open aazizisoufiane opened 5 months ago

aazizisoufiane commented 5 months ago

Issue: Encountering botocore.errorfactory.ThrottlingException when using invoke_model due to rate limits. AWS recommends exponential backoff for API retries.

Current Function: def _call_model(self, body: str) -> str: return self.predictor.invoke_model(modelId=self._model_name, body=body, accept="application/json", contentType="application/json")

Question: Would a PR implementing exponential backoff with the backoff library for this operation be welcome? Keen to contribute a solution to improve request reliability.

arnavsinghvi11 commented 4 months ago

Hi @aazizisoufiane , a PR would definitely be welcome! Just curious though, which AWS model were you using for this? I'm assuming Anthropic since I only see that AWS LM configured with backoff but I know @drawal1 is working on a AWS LM refactor in #795 so feel free to check out if it's handled there or if it needs to be added.

drawal1 commented 4 months ago

@arnavsinghvi11 PR #795 does not implement any special code for exponential backoff so it would be welcome. @aazizisoufiane can implement once PR #795 is merged.

just noticed there is a dsp/modules/aws_lm.py file that will also be obsolete after PR #795. Deleting it so there's no confusion...