search

Azure-Samples / private-openai-with-apim-for-chargeback

Open AI with Private Endpoints behind APIM and functionality to get tokens consumption for each consumer
MIT License
27 stars 13 forks source link

Feature request - Multiple OpenAi endpoints as well as integration with linked MS failover policy. #7

Open stateofthearb opened 1 month ago

stateofthearb commented 1 month ago

Please provide us with the following information:

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [x] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Any log messages given by the failure

Expected/desired behavior

It would be good to see this available for multiple endpoints, I appreciate the work done on this project but in reality most production workloads will be running multiple OpenAI instances with failover between PTU & PAYG. I also appreciate that MS have different teams developing different features but it would be great if this project could be integrated with https://github.com/Azure-Samples/openai-apim-lb/tree/main.

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)

Versions

Mention any other details that might be useful

Thanks! We'll be in touch soon.