ray-project / llmperf

LLMPerf is a library for validating and benchmarking LLMs
Apache License 2.0
659 stars 113 forks source link

Added Azure OpenAI endpoint support #41

Open datlife opened 8 months ago

datlife commented 8 months ago

Added Azure OpenAI endpoint support. For Azure OpenAI, users need to specify OPENAI_API_VERSION and update the header to api-key: <key> instead of Authorization: Bearer <token>

Reference: https://learn.microsoft.com/en-us/azure/ai-services/openai/reference#chat-completions

Testing: I was able top use llmperf for Azure endpoint

# Azure openAI
export OPENAI_API_BASE="https://<resporce_name>.openai.azure.com/openai/deployments/gpt-3.5-turbo"
export OPENAI_API_KEY=<azure_key_from_resource_name>
export OPENAI_API_VERSION="2023-03-15-preview"

python token_benchmark_ray.py \
--model "gpt-3.5-turbo" \
--mean-input-tokens 550 \
--stddev-input-tokens 150 \
--mean-output-tokens 150 \
--stddev-output-tokens 10 \
--timeout 600 \
--results-dir "result_outputs" \
--llm-api openai%
NicolasGrosjeanProbayes commented 7 months ago

Thanks a lot @datlife, it saved me a lot of time