Open datlife opened 8 months ago
Added Azure OpenAI endpoint support. For Azure OpenAI, users need to specify OPENAI_API_VERSION and update the header to api-key: <key> instead of Authorization: Bearer <token>
OPENAI_API_VERSION
api-key: <key>
Authorization: Bearer <token>
Reference: https://learn.microsoft.com/en-us/azure/ai-services/openai/reference#chat-completions
Testing: I was able top use llmperf for Azure endpoint
# Azure openAI export OPENAI_API_BASE="https://<resporce_name>.openai.azure.com/openai/deployments/gpt-3.5-turbo" export OPENAI_API_KEY=<azure_key_from_resource_name> export OPENAI_API_VERSION="2023-03-15-preview" python token_benchmark_ray.py \ --model "gpt-3.5-turbo" \ --mean-input-tokens 550 \ --stddev-input-tokens 150 \ --mean-output-tokens 150 \ --stddev-output-tokens 10 \ --timeout 600 \ --results-dir "result_outputs" \ --llm-api openai%
Thanks a lot @datlife, it saved me a lot of time
Added Azure OpenAI endpoint support. For Azure OpenAI, users need to specify
OPENAI_API_VERSION
and update the header toapi-key: <key>
instead ofAuthorization: Bearer <token>
Reference: https://learn.microsoft.com/en-us/azure/ai-services/openai/reference#chat-completions
Testing: I was able top use llmperf for Azure endpoint