Azure-Samples / apim-genai-gateway-toolkit

A repo to accelerate development and testing of GenAI Gateways built with Azure API Management. Includes various capabilities as examples/starters.
MIT License
17 stars 6 forks source link

Add retry to manage spikes v2 #29

Open lucashuet93 opened 2 weeks ago

lucashuet93 commented 2 weeks ago

Please provide us with the following information:

This issue is for a: (mark with an x)

- [x] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

load test the manage-spikes-with-payg-v2 and observe 429s returned to the user

Expected/desired behavior

callers should be shielded from 429s returned from backend APIs via retry policy (APIM should auto retry)

Mention any other details that might be useful

We may want to merge the v1 and v2 approaches here