Using structured output with vision models like gpt-4o-mini works. I'd like to do the same for Llama-3.2-11B-Vision-Instruct from GitHub models. Currently it throws an exception.
Set modelName in Program.cs to Llama-3.2-11B-Vision-Instruct
Run the application
Working Azure AI Inference SDK Code
Replacing the inference loop with this code works
foreach(var path in filePaths)
{
var file = await File.ReadAllBytesAsync(path);
var messages = new List<ChatRequestMessage>
{
new ChatRequestSystemMessage(systemPrompt),
new ChatRequestUserMessage(new ChatMessageImageContentItem(BinaryData.FromBytes(file), "image/jpeg")),
new ChatRequestUserMessage("Extract the marked passages from the image")
};
var response = await aoaiClient.CompleteAsync(new ChatCompletionsOptions{
Messages = messages,
Model = modelName
});
Console.WriteLine(response.Value.Content);
}
Expected Output
The marked passages are:
* If we try ten experiments and none of them work, we have a choice. We can take it personally, and think of ourselves as a failure and question our ability to solve the problems that come our way.
* We are required to believe in something that doesn't exist in order to allow it to come into being.
* Faith is rewarded, perhaps even more than talent or ability.
These passages are found on pages 278 and 279.
Similarly, using MEAI without structured output
foreach(var path in filePaths)
{
var file = await File.ReadAllBytesAsync(path);
var messages = new List<ChatMessage>
{
new ChatMessage(ChatRole.System, systemPrompt),
new ChatMessage(ChatRole.User, new AIContent[] {
new ImageContent(file, "image/jpeg"),
new TextContent("Extract the marked passages from the image"),
})
};
var response = await meaiClient.CompleteAsync(messages, options: new ChatOptions {Temperature = 0.1f});
Console.WriteLine(response.Message);
}
Expected behavior
Response returns LLM response as specified type parameter, even if it's not supported by the model.
Unhandled exception. Azure.RequestFailedException: An error occurred during the request processing: All connection attempts failed | Type: ConnectError | Traceback (most recent call last):|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions|| yield|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_transports/default.py", line 366, in handle_async_request|| resp = await self._pool.handle_async_request(req)|| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 216, in handle_async_request|| raise exc from None|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 196, in handle_async_request|| response = await connection.handle_async_request(|| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_async/connection.py", line 99, in handle_async_request|| raise exc|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_async/connection.py", line 76, in handle_async_request|| stream = await self._connect(request)|| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_async/connection.py", line 122, in _connect|| stream = await self._network_backend.connect_tcp(**kwargs)|| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_backends/auto.py", line 30, in connect_tcp|| return await self._backend.connect_tcp(|| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 115, in connect_tcp|| with map_exceptions(exc_map):|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/contextlib.py", line 158, in __exit__|| self.gen.throw(typ, value, traceback)|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions|| raise to_exc(exc) from exc||httpcore.ConnectError: All connection attempts failed||||The above exception was the direct cause of the following exception:||||Traceback (most recent call last):|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__|| await self.app(scope, receive, _send)|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/billingproxy/main.py", line 411, in __call__|| return await handler_task|| ^^^^^^^^^^^^^^^^^^|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__|| with collapse_excgroups():|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/contextlib.py", line 158, in __exit__|| self.gen.throw(typ, value, traceback)|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups|| raise exc|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__|| response = await self.dispatch_func(request, call_next)|| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/billingproxy/main.py", line 377, in dispatch|| response = await call_next(request)|| ^^^^^^^^^^^^^^^^^^^^^^^^|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py", line 165, in call_next|| raise app_exc|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py", line 151, in coro|| await self.app(scope, receive_or_disconnect, send_no_error)|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__|| with collapse_excgroups():|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/contextlib.py", line 158, in __exit__|| self.gen.throw(typ, value, traceback)|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups|| raise exc|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__|| response = await self.dispatch_func(request, call_next)|| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/billingproxy/main.py", line 353, in dispatch|| response = await call_next(request)|| ^^^^^^^^^^^^^^^^^^^^^^^^|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py", line 165, in call_next|| raise app_exc|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py", line 151, in coro|| await self.app(scope, receive_or_disconnect, send_no_error)|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__|| await self.app(scope, receive, send)|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__|| await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app|| raise exc|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app|| await app(scope, receive, sender)|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__|| await self.middleware_stack(scope, receive, send)|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/routing.py", line 776, in app|| await route.handle(scope, receive, send)|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle|| await self.app(scope, receive, send)|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/routing.py", line 77, in app|| await wrap_app_handling_exceptions(app, request)(scope, receive, send)|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app|| raise exc|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app|| await app(scope, receive, sender)|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/routing.py", line 72, in app|| response = await func(request)|| ^^^^^^^^^^^^^^^^^^^|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app|| raw_response = await run_endpoint_function(|| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function|| return await dependant.call(**values)|| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/billingproxy/apis/vllm/chat.py", line 322, in chat|| response = await downstream_client_getter().send(downstream_request)||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_client.py", line 1617, in send|| response = await self._send_handling_auth(|| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_client.py", line 1645, in _send_handling_auth|| response = await self._send_handling_redirects(|| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects|| response = await self._send_single_request(request)|| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_client.py", line 1719, in _send_single_request|| response = await transport.handle_async_request(request)|| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_transports/default.py", line 365, in handle_async_request|| with map_httpcore_exceptions():|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/contextlib.py", line 158, in __exit__|| self.gen.throw(typ, value, traceback)|| File "/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_transports/default.py", line 83, in map_httpcore_exceptions|| raise mapped_exc(message) from exc||httpx.ConnectError: All connection attempts failed||
Status: 500 (Internal Server Error)
ErrorCode: Internal Server Error
Content:
{"error":{"code":"Internal Server Error","message":"An error occurred during the request processing: All connection attempts failed | Type: ConnectError | Traceback (most recent call last):|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_transports/default.py\", line 66, in map_httpcore_exceptions|| yield|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_transports/default.py\", line 366, in handle_async_request|| resp = await self._pool.handle_async_request(req)|| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_async/connection_pool.py\", line 216, in handle_async_request|| raise exc from None|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_async/connection_pool.py\", line 196, in handle_async_request|| response = await connection.handle_async_request(|| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_async/connection.py\", line 99, in handle_async_request|| raise exc|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_async/connection.py\", line 76, in handle_async_request|| stream = await self._connect(request)|| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_async/connection.py\", line 122, in _connect|| stream = await self._network_backend.connect_tcp(**kwargs)|| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_backends/auto.py\", line 30, in connect_tcp|| return await self._backend.connect_tcp(|| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_backends/anyio.py\", line 115, in connect_tcp|| with map_exceptions(exc_map):|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/contextlib.py\", line 158, in __exit__|| self.gen.throw(typ, value, traceback)|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpcore/_exceptions.py\", line 14, in map_exceptions|| raise to_exc(exc) from exc||httpcore.ConnectError: All connection attempts failed||||The above exception was the direct cause of the following exception:||||Traceback (most recent call last):|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/errors.py\", line 164, in __call__|| await self.app(scope, receive, _send)|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/billingproxy/main.py\", line 411, in __call__|| return await handler_task|| ^^^^^^^^^^^^^^^^^^|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py\", line 189, in __call__|| with collapse_excgroups():|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/contextlib.py\", line 158, in __exit__|| self.gen.throw(typ, value, traceback)|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/_utils.py\", line 93, in collapse_excgroups|| raise exc|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py\", line 191, in __call__|| response = await self.dispatch_func(request, call_next)|| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/billingproxy/main.py\", line 377, in dispatch|| response = await call_next(request)|| ^^^^^^^^^^^^^^^^^^^^^^^^|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py\", line 165, in call_next|| raise app_exc|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py\", line 151, in coro|| await self.app(scope, receive_or_disconnect, send_no_error)|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py\", line 189, in __call__|| with collapse_excgroups():|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/contextlib.py\", line 158, in __exit__|| self.gen.throw(typ, value, traceback)|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/_utils.py\", line 93, in collapse_excgroups|| raise exc|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py\", line 191, in __call__|| response = await self.dispatch_func(request, call_next)|| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/billingproxy/main.py\", line 353, in dispatch|| response = await call_next(request)|| ^^^^^^^^^^^^^^^^^^^^^^^^|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py\", line 165, in call_next|| raise app_exc|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/base.py\", line 151, in coro|| await self.app(scope, receive_or_disconnect, send_no_error)|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/cors.py\", line 85, in __call__|| await self.app(scope, receive, send)|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/middleware/exceptions.py\", line 65, in __call__|| await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/_exception_handler.py\", line 64, in wrapped_app|| raise exc|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/_exception_handler.py\", line 53, in wrapped_app|| await app(scope, receive, sender)|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/routing.py\", line 756, in __call__|| await self.middleware_stack(scope, receive, send)|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/routing.py\", line 776, in app|| await route.handle(scope, receive, send)|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/routing.py\", line 297, in handle|| await self.app(scope, receive, send)|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/routing.py\", line 77, in app|| await wrap_app_handling_exceptions(app, request)(scope, receive, send)|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/_exception_handler.py\", line 64, in wrapped_app|| raise exc|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/_exception_handler.py\", line 53, in wrapped_app|| await app(scope, receive, sender)|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/starlette/routing.py\", line 72, in app|| response = await func(request)|| ^^^^^^^^^^^^^^^^^^^|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/fastapi/routing.py\", line 278, in app|| raw_response = await run_endpoint_function(|| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/fastapi/routing.py\", line 191, in run_endpoint_function|| return await dependant.call(**values)|| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/billingproxy/apis/vllm/chat.py\", line 322, in chat|| response = await downstream_client_getter().send(downstream_request)|| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_client.py\", line 1617, in send|| response = await self._send_handling_auth(|| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_client.py\", line 1645, in _send_handling_auth|| response = await self._send_handling_redirects(|| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_client.py\", line 1682, in _send_handling_redirects|| response = await self._send_single_request(request)|| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_client.py\", line 1719, in _send_single_request|| response = await transport.handle_async_request(request)|| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_transports/default.py\", line 365, in handle_async_request|| with map_httpcore_exceptions():|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/contextlib.py\", line 158, in __exit__|| self.gen.throw(typ, value, traceback)|| File \"/opt/microsoft-maas/miniconda/envs/proxy/lib/python3.11/site-packages/httpx/_transports/default.py\", line 83, in map_httpcore_exceptions|| raise mapped_exc(message) from exc||httpx.ConnectError: All connection attempts failed||","status":500}}
Headers:
Date: Mon, 04 Nov 2024 23:54:21 GMT
Transfer-Encoding: chunked
Connection: keep-alive
azureml-model-group: REDACTED
request-context: REDACTED
x-ms-rai-invoked: REDACTED
X-Request-ID: REDACTED
ms-azureml-model-error-reason: REDACTED
ms-azureml-model-error-statuscode: REDACTED
x-ms-client-request-id: 09647581-2868-4eef-b19d-be48a4c54033
azureml-model-deployment: REDACTED
azureml-model-session: REDACTED
Strict-Transport-Security: REDACTED
X-Content-Type-Options: REDACTED
x-aml-cluster: REDACTED
x-request-time: REDACTED
Content-Type: application/json
at Azure.Core.HttpPipelineExtensions.ProcessMessageAsync(HttpPipeline pipeline, HttpMessage message, RequestContext requestContext, CancellationToken cancellationToken)
at Azure.AI.Inference.ChatCompletionsClient.CompleteAsync(RequestContent content, String extraParams, RequestContext context)
at Azure.AI.Inference.ChatCompletionsClient.CompleteAsync(ChatCompletionsOptions chatCompletionsOptions, CancellationToken cancellationToken)
at Microsoft.Extensions.AI.AzureAIInferenceChatClient.CompleteAsync(IList`1 chatMessages, ChatOptions options, CancellationToken cancellationToken)
at Microsoft.Extensions.AI.ChatClientStructuredOutputExtensions.CompleteAsync[T](IChatClient chatClient, IList`1 chatMessages, JsonSerializerOptions serializerOptions, ChatOptions options, Nullable`1 useNativeJsonSchema, CancellationToken cancellationToken)
at Program.<Main>$(String[] args) in C:\Dev\AIBookmarks\Program.cs:line 76
at Program.<Main>(String[] args)
stephentoub
commented
2 hours ago
Description
Using structured output with vision models like gpt-4o-mini works. I'd like to do the same for Llama-3.2-11B-Vision-Instruct from GitHub models. Currently it throws an exception.
Reproduction Steps
modelNamein Program.cs toLlama-3.2-11B-Vision-InstructWorking Azure AI Inference SDK Code
Replacing the inference loop with this code works
Expected Output
Similarly, using MEAI without structured output
Expected behavior
Response returns LLM response as specified type parameter, even if it's not supported by the model.
See generic completion API for reference.
Actual behavior
Exception is thrown.
Regression?
No response
Known Workarounds
No response
Configuration
No response
Other information
No response