Unable to extract values for util_avg and util_95th parameters using the tool

[suryakurapati]

Whenever the code is being run with below set off arguments, all the output fields are being extracted except util_avg and util_95th.

Run Commad: python -m benchmark.bench load --api-version --api-key-env --clients 10 --shape-profile custom --context-tokens 4000 --max-tokens 5500 --deployment gpt-4-8k-0613 --duration 60 --requests 60 --output-format human

I see the logic for util_avg and util_95th in the code as below:

step1: Get the response of the input request

• response = await session.post(self.url, headers=headers, json=body)

step2: Transformation and logic - part 1

• UTILIZATION_HEADER = "azure-openai-deployment-utilization"
• util_str = response.headers[UTILIZATION_HEADER]
• stats.deployment_utilization = float(util_str[:-1]) Note: In this step, I do not see response.headers having any "azure-openai-deployment-utilization" key in it. Since this key is not found in response.headers, in the next step the final list is found to be empty list and getting the final result as n/a

step3: Transformation and logic - part 2

• self.utilizations._append(stats.request_start_time, stats.deployment_utilization)
• util_avg = f"{round(np.average(self.utilizations._values()), 1)}%" if self.utilizations._len() > 0 else "n/a" util_95th = f"{round(np.percentile(self.utilizations._values(), 95), 1)}%" if self.utilizations._len() > 1 else "n/a"

Could you take a look and confirm if any update to the code is required in order to extract util_avg and util_95th parameters.

[technicianted]

Utilization stats are only available for managed-provisioned deployment endpoints. If you do not see the header then chances are your endpoint isn't managed-provisioned.

[technicianted]

Closing this as Azure support issue. Please reopen if otherwise.