[Bug]: Undefined array key "usage" with Vision endpoint on Azure

[pimverbij]

Description

I'm trying to use vision on Azure, this was working before.

It seems that Azure has changed the format of their response for the Vision endpoint.

I am creating the client with the factory etc. But since today the code that had been working has stopped working. This results in the error Undefined array key "usage" I'm running the latests version of the client.

Steps To Reproduce

         $baseUri = 'https://tellet-azure-sweden.openai.azure.com/openai/deployments/tellet-gpt-4-vision-preview';

        $client = OpenAI::factory()
        ->withBaseUri($baseUri)
        ->withHttpClient(new \GuzzleHttp\Client(['timeout' => 300, 'connect_timeout' => 300]))
        ->withHttpHeader('api-key', $apiKey)
        ->withQueryParam('api-version', '2023-12-01-preview')
        ->make();

         $base64Image = $this->imageToBase64($image);
         $encoded = 'data:image/jpeg;base64,'.$base64Image;

    $imageArray[] = [
        'type' => 'text',
        'text' => 'Analyse this photograph and provide a detailed description of its contents. Focus on identifying and listing all visible items, with particular attention to food items present. '
    ];

    $imageArray[] = [
        'type' => 'image_url',
        'image_url' => ['url' => $encoded]
    ];

    $messagesArray = array();
    $messagesArray[] = [
        'role' => 'user',
        'content' => $imageArray
    ];

    $settings['messages'] = $messagesArray;
    $settings['max_tokens'] = 1200;
    $settings['messages'] = $messagesArray;
    $response = $client->chat()->create($settings);

OpenAI PHP Client Version

v0.8.4

PHP Version

8.1

Notes

No response

[marcocanestrari]

Possibly connected to https://github.com/openai-php/client/issues/336

[pimverbij]

Doing some more digging and it looks like this specific model is not returning the "usage"

Response from the gpt-4-vision-preview model with an image OpenAI\ValueObjects\Transporter\Response Object ( [data:OpenAI\ValueObjects\Transporter\Response:private] => Array ( [choices] => Array ( [0] => Array ( [content_filter_results] => Array ( [hate] => Array ( [filtered] => [severity] => safe ) [self_harm] => Array ( [filtered] => [severity] => safe ) [sexual] => Array ( [filtered] => [severity] => safe ) [violence] => Array ( [filtered] => [severity] => safe ) ) [finish_reason] => stop [index] => 0 [message] => Array ( [content] => The photograph showcases a black and white scene with two individuals positioned side by side in contrasting postures. The person on the left appears to be palm-outward in a gesture of stopping or "no entry," and wearing a short-sleeved, casual top. The individual on the right is extending their hand, palm-up towards the camera in a gesture that might suggest offering, asking, or reaching out. The background of the image is out of focus, but a window or a light source is noticeable behind the two individuals. After analyzing the image, there appear to be no identifiable food items present. The photograph is solely focused on capturing the gestures and expressions of the individuals with no visible consumables or other specific objects. [role] => assistant ) ) ) [created] => 0 [id] => [model] => [object] => chat.completion [prompt_filter_results] => Array ( [0] => Array ( [prompt_index] => 0 [content_filter_results] => Array ( [hate] => Array ( [filtered] => [severity] => safe ) [self_harm] => Array ( [filtered] => [severity] => safe ) [sexual] => Array ( [filtered] => [severity] => safe ) [violence] => Array ( [filtered] => [severity] => safe ) ) ) ) ) [meta:OpenAI\ValueObjects\Transporter\Response:private] => OpenAI\Responses\Meta\MetaInformation Object ( [requestId] => 218dc1a4-03ff-45ef-a191-69efe45fbde2 [openai] => OpenAI\Responses\Meta\MetaInformationOpenAI Object ( [model] => [organization] => [version] => [processingMs] => ) [requestLimit] => OpenAI\Responses\Meta\MetaInformationRateLimit Object ( [limit] => [remaining] => 28 [reset] => ) [tokenLimit] => OpenAI\Responses\Meta\MetaInformationRateLimit Object ( [limit] => [remaining] => 25314 [reset] => ) ) ) Response from the gpt-4-vision-preview model with text OpenAI\ValueObjects\Transporter\Response Object ( [data:OpenAI\ValueObjects\Transporter\Response:private] => Array ( [choices] => Array ( [0] => Array ( [content_filter_results] => Array ( [hate] => Array ( [filtered] => [severity] => safe ) [self_harm] => Array ( [filtered] => [severity] => safe ) [sexual] => Array ( [filtered] => [severity] => safe ) [violence] => Array ( [filtered] => [severity] => safe ) ) [finish_reason] => stop [index] => 0 [message] => Array ( [content] => "Darwin's Origin of Species" is a groundbreaking work where Charles Darwin introduces the theory of natural selection. He argues that species evolve over time through a process of inherited variations that enhance an organism's ability to survive and reproduce. Darwin's observations from his voyage on the HMS Beagle, particularly in the Galápagos Islands, support his conclusion that the diversity of life is a result of modifications over generations. The book challenges traditional views by suggesting common descent among all organisms and outlining the evolutionary process that has shaped the natural world, laying the foundation for modern evolutionary biology. [role] => assistant ) ) ) [created] => 0 [id] => [model] => [object] => chat.completion [prompt_filter_results] => Array ( [0] => Array ( [prompt_index] => 0 [content_filter_results] => Array ( [hate] => Array ( [filtered] => [severity] => safe ) [self_harm] => Array ( [filtered] => [severity] => safe ) [sexual] => Array ( [filtered] => [severity] => safe ) [violence] => Array ( [filtered] => [severity] => safe ) ) ) ) ) [meta:OpenAI\ValueObjects\Transporter\Response:private] => OpenAI\Responses\Meta\MetaInformation Object ( [requestId] => 07bbc9e7-2658-44af-b0e2-7ab1666ab756 [openai] => OpenAI\Responses\Meta\MetaInformationOpenAI Object ( [model] => [organization] => [version] => [processingMs] => ) [requestLimit] => OpenAI\Responses\Meta\MetaInformationRateLimit Object ( [limit] => [remaining] => 29 [reset] => ) [tokenLimit] => OpenAI\Responses\Meta\MetaInformationRateLimit Object ( [limit] => [remaining] => 28772 [reset] => ) ) )

Response from GPT-4 model text OpenAI\ValueObjects\Transporter\Response Object ( [data:OpenAI\ValueObjects\Transporter\Response:private] => Array ( [id] => chatcmpl-8sBDBfLcGMOe2UwpUQuMHMKfYgrNI [object] => chat.completion [created] => 1707923501 [model] => gpt-4 [prompt_filter_results] => Array ( [0] => Array ( [prompt_index] => 0 [content_filter_results] => Array ( [hate] => Array ( [filtered] => [severity] => safe ) [self_harm] => Array ( [filtered] => [severity] => safe ) [sexual] => Array ( [filtered] => [severity] => safe ) [violence] => Array ( [filtered] => [severity] => safe ) ) ) ) [choices] => Array ( [0] => Array ( [finish_reason] => stop [index] => 0 [message] => Array ( [role] => assistant [content] => "Origin of Species" by Charles Darwin is a revolutionary work that discusses the theory of evolution: life's diversity is a result of gradual changes over many generations. Darwin's observations from the Galapagos Islands led to his groundbreaking notion of natural selection, stating individuals in a species show wide range of variation, and those with characteristics best adapted to their environment have higher survival chances. This book revolutionized the scientific understanding of life's development, refuting the belief that species were unchanging, and instead suggesting a branching pattern of evolution, undergirded by common ancestry. ) [content_filter_results] => Array ( [hate] => Array ( [filtered] => [severity] => safe ) [self_harm] => Array ( [filtered] => [severity] => safe ) [sexual] => Array ( [filtered] => [severity] => safe ) [violence] => Array ( [filtered] => [severity] => safe ) ) ) ) [usage] => Array ( [prompt_tokens] => 34 [completion_tokens] => 114 [total_tokens] => 148 ) ) [meta:OpenAI\ValueObjects\Transporter\Response:private] => OpenAI\Responses\Meta\MetaInformation Object ( [requestId] => f1b6bdec-746d-44ad-a92f-49278bf1456c [openai] => OpenAI\Responses\Meta\MetaInformationOpenAI Object ( [model] => [organization] => [version] => [processingMs] => ) [requestLimit] => OpenAI\Responses\Meta\MetaInformationRateLimit Object ( [limit] => [remaining] => 39 [reset] => ) [tokenLimit] => OpenAI\Responses\Meta\MetaInformationRateLimit Object ( [limit] => [remaining] => 38800 [reset] => ) ) )