SK + Phi-3-vision-128k-instruct-onnx-cpu => Not getting Results

[aherrick]

Using the following local Model pulled from HF here is my code:

        var modelPath =
            @"C:\models\Phi-3-vision-128k-instruct-onnx-cpu\cpu-int4-rtn-block-32-acc-level-4";

#pragma warning disable SKEXP0070 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.

        // create kernel
        var kernel = Kernel
            .CreateBuilder()
            .AddOnnxRuntimeGenAIChatCompletion(
                modelId: "microsoft/Phi-3-vision-128k-instruct",
                //modelId: "Phi-3-vision-128k-instruct-onnx-cp",
                modelPath: modelPath
            )
            .Build();

        // create chat
        var chat = kernel.GetRequiredService<IChatCompletionService>();
        var history = new ChatHistory();

        var testImgPath = Path.Combine(Directory.GetCurrentDirectory(), "imgs", "test.png");

        // create chat collection items
        var collectionItems = new ChatMessageContentItemCollection
        {
            new TextContent("What is the image?"),
            new ImageContent(File.ReadAllBytes(testImgPath), "image/png")
        };
        history.AddUserMessage(collectionItems);

        Console.Write($"Phi3: ");
        var result = await chat.GetChatMessageContentsAsync(history);
        Console.WriteLine(result[^1].Content);

        Console.Read();

I keep getting responses like this: (image below)

 Sorry, I cannot answer this question.

The image is not shown. This is a placeholder for a content that cannot be displayed.</s>

Is there something wrong with my code?

[leestott]

@aherrick

The error message you're seeing suggests that the image content is not being properly processed or displayed. Here are a few potential issues and solutions:

1. File Path Issue

• Check the Path: Ensure that the path to the image file is correct and that the file exists at that location.
• Permissions: Verify that your application has the necessary permissions to read the file.

2. File Reading Issue

• File Format: Ensure the image file is in the correct format (e.g., PNG) and not corrupted.
• File Reading Code: Double-check the code that reads the file to ensure it's correct.

3. Service Configuration

• Model Configuration: Ensure that the model and service are correctly configured to handle image inputs.
• Dependencies: Verify that all necessary dependencies and libraries are correctly installed and referenced.

4. Error Handling

• Exception Handling: Add try-catch blocks around the code that processes the image to catch and log any exceptions.

Example Code with Error Handling

Here's an example of how you might add error handling to your code:

try
{
    var testImgPath = Path.Combine(Directory.GetCurrentDirectory(), "imgs", "test.png");

    if (!File.Exists(testImgPath))
    {
        Console.WriteLine("Image file not found.");
        return;
    }

    var collectionItems = new ChatMessageContentItemCollection
    {
        new TextContent("What is the image?"),
        new ImageContent(File.ReadAllBytes(testImgPath), "image/png")
    };
    history.AddUserMessage(collectionItems);

    Console.Write($"Phi3: ");
    var result = await chat.GetChatMessageContentsAsync(history);
    Console.WriteLine(result.Content);
}
catch (Exception ex)
{
    Console.WriteLine($"An error occurred: {ex.Message}");
}

Debugging Steps

1. Log File Paths and Status: Print out the file path and check if the file exists.
2. Check Image Content: Ensure the image content is correctly read and not null or empty.
3. Service Logs: Check logs from the AI service to see if there are any errors or warnings related to image processing.

Your code looks mostly valid, but there are a few things to check and consider:

1. Namespace and Usings: Ensure you have the necessary namespaces and using directives at the top of your file.
2. Async Context: Since you're using await, make sure your method is marked as async.
3. Error Handling: Consider adding error handling to manage potential exceptions.

Here's a revised version with these considerations:

using System;
using System.IO;
using Microsoft.Extensions.DependencyInjection;
using YourNamespace.Services; // Replace with actual namespace for IChatCompletionService and other services

public class Program
{
    public static async Task Main(string[] args)
    {
        var modelPath = @"C:\models\Phi-3-vision-128k-instruct-onnx-cpu\cpu-int4-rtn-block-32-acc-level-4";

        #pragma warning disable SKEXP0070 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.

        // create kernel
        var kernel = Kernel
            .CreateBuilder()
            .AddOnnxRuntimeGenAIChatCompletion(
                modelId: "microsoft/Phi-3-vision-128k-instruct",
                //modelId: "Phi-3-vision-128k-instruct-onnx-cp",
                modelPath: modelPath
            )
            .Build();

        // create chat
        var chat = kernel.GetRequiredService<IChatCompletionService>();
        var history = new ChatHistory();

        var testImgPath = Path.Combine(Directory.GetCurrentDirectory(), "imgs", "test.png");

        // create chat collection items
        var collectionItems = new ChatMessageContentItemCollection
        {
            new TextContent("What is the image?"),
            new ImageContent(File.ReadAllBytes(testImgPath), "image/png")
        };
        history.AddUserMessage(collectionItems);

        Console.Write($"Phi3: ");
        var result = await chat.GetChatMessageContentsAsync(history);
        Console.WriteLine(result.Content);

        Console.Read();
    }
}

[kinfey]

Can you try using ONNX as service ? Maybe you can use Hugging face connector in SK

[leestott]

@aherrick you can see an example of how to use the Hugging Face Connector in Semantic Kernel at https://devblogs.microsoft.com/semantic-kernel/how-to-use-hugging-face-models-with-semantic-kernel/