Lucashh/22 build features

Azure-Samples / apim-genai-gateway-toolkit

A repo to accelerate development and testing of GenAI Gateways built with Azure API Management. Includes various capabilities as examples/starters.

MIT License

35 stars 12 forks source link

Lucashh/22 build features #26

Closed lucashuet93 closed 3 months ago

lucashuet93 commented 3 months ago

adds features released at Build
- adds new simple-round-robin-v2 API that utilizes load balancing infrastructure
- adds new weighted-round-robin-v2 API that utilizes load balancing infrastructure
- adds new retry-with-payg-v2 API that utilizes load balancing and circuit breaker infrastructure
- updates rate-limiting API to utilize <azure-openai-token-limit /> policy
- updates usage-tracking API to utilize <azure-openai-emit-token-metric /> policy
adds e2e tests for the v2 APIs
updates docs

The usage-tracking e2e test still needs to display the token metrics, but I elected to leave that for #5.

stuartleeks commented 3 months ago

@lucashuet93 I tried deploying this but got Cannot create API 'aoai-api-rate-limiting' with the same Path 'rate-limiting/openai' as API 'aoai-api-rate-limting' unless it's a part of the same version set. Is there a clash with previous versions?

stuartleeks commented 3 months ago

@lucashuet93 can you also add README files for the v2 capabilities?

lucashuet93 commented 3 months ago

@stuartleeks I fixed a typo in the rate limting APIs path so now it needs to be freshly redeployed or you'll get the above error. I will add READMEs for them but thought I might do so in a separate docs related PR. Can add here.

lucashuet93 commented 3 months ago

@stuartleeks I added the docs for the new capabilities and updated the existing ones. Ready for re-review.

stuartleeks commented 3 months ago

@stuartleeks I fixed a typo in the rate limting APIs path so now it needs to be freshly redeployed or you'll get the above error.

Is it worth adding a troubleshooting section to the main README? That would give us somewhere that we can include notes like this

lucashuet93 commented 3 months ago

@stuartleeks I fixed a typo in the rate limting APIs path so now it needs to be freshly redeployed or you'll get the above error.

Is it worth adding a troubleshooting section to the main README? That would give us somewhere that we can include notes like this

Good idea, added!