Prompt logging monitoring

[bmis]

Does this solution have the ability to log prompts going to Azure Open AI and responses from Azure Open AI? For example using Azure Monitor?

Thanks.

[andredewes]

Not yet. Meanwhile, one idea would be to have APIM fronting this load balancer and use its logging capabilities. There is one Azure reference architecture for this: https://learn.microsoft.com/en-us/azure/architecture/ai-ml/openai/architecture/log-monitor-azure-openai

[azureholic]

for a logging example (in this case token usage) have a look at this repo https://github.com/Azure/enterprise-azureai

@andredewes : I was also on the path of using YARP for monitoring and took the liberty to use your priority-loadbalancing code. Thanks for the great work!