[BUG]: System.NullReferenceException in ChatSession.ChatAsync upgrading to 0.18

[grammophone]

Description

I just upgraded the NuGet packages from 0.16 to 0.18. CPU backend. The simple test console program still compiled but it no longer runs. Inside ChatSession.ChatAsync I get the exception:

System.NullReferenceException: 'Object reference not set to an instance of an object.'

LLama.InteractiveExecutor.InferInternal(LLama.Abstractions.IInferenceParams, LLama.StatefulExecutorBase.InferStateArgs)
LLama.StatefulExecutorBase.InferAsync(string, LLama.Abstractions.IInferenceParams, System.Threading.CancellationToken)
LLama.LLamaTransforms.KeywordTextOutputStreamTransform.TransformAsync(System.Collections.Generic.IAsyncEnumerable<string>)
LLama.LLamaTransforms.KeywordTextOutputStreamTransform.TransformAsync(System.Collections.Generic.IAsyncEnumerable<string>)
LLama.ChatSession.ChatAsyncInternal(string, LLama.Abstractions.IInferenceParams, System.Threading.CancellationToken)
LLama.ChatSession.ChatAsyncInternal(string, LLama.Abstractions.IInferenceParams, System.Threading.CancellationToken)
LLama.ChatSession.ChatAsync(LLama.Common.ChatHistory.Message, bool, LLama.Abstractions.IInferenceParams, System.Threading.CancellationToken)
LLama.ChatSession.ChatAsync(LLama.Common.ChatHistory.Message, bool, LLama.Abstractions.IInferenceParams, System.Threading.CancellationToken)
LLamaSharpConsole.Program.Main(string[]) in Program.cs

Reproduction Steps

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using LLama;
using LLama.Common;
using LLama.Sampling;

namespace LLamaSharpConsole
{
    internal class Program
    {
        static async Task Main(string[] args)
        {
            string modelPath = @"D:\Projects\dotNet\Solutions\LLMPlayground\Models\model.gguf";
            //string modelPath = @"D:\Projects\dotNet\Solutions\LLMPlayground\Models\model.quantized.gguf";

            var parameters = new ModelParams(modelPath)
            {
                ContextSize = 4096,
                GpuLayerCount = 10
            };
            using var model = LLamaWeights.LoadFromFile(parameters);
            using var context = model.CreateContext(parameters);
            var executor = new InteractiveExecutor(context);

            ChatHistory chatHistory = new ChatHistory();

            ChatSession session = new(executor, chatHistory);

            session.WithOutputTransform(new LLamaTransforms.KeywordTextOutputStreamTransform(
                    new string[] { "User:", "Assistant:", "�" },
                    redundancyLength: 8));

            InferenceParams inferenceParams = new InferenceParams()
            {
                AntiPrompts = new List<string> { "User:", "Assistant:" },
                //SamplingPipeline = new Mirostat2SamplingPipeline()
                //{
                //  Tau = 1f,
                //  Eta = 0.2f
                //}
            };

            Console.ForegroundColor = ConsoleColor.Yellow;
            Console.WriteLine("The chat session has started.");
            Console.Write("\r\nReady> ");

            // show the prompt
            Console.ForegroundColor = ConsoleColor.Green;
            string userInput = Console.ReadLine() ?? "";

            while (userInput != "exit")
            {
                Console.ForegroundColor = ConsoleColor.White;

                await foreach (
                        var text
                        in session.ChatAsync(
                                new ChatHistory.Message(AuthorRole.User, userInput + "\nAssistant: "),
                                inferenceParams))
                {
                    Console.Write(text);
                }

                Console.ForegroundColor = ConsoleColor.Yellow;
                Console.Write("\r\nReady> ");

                Console.ForegroundColor = ConsoleColor.Green;
                userInput = Console.ReadLine() ?? "";
            }

            Console.ForegroundColor = ConsoleColor.White;
        }

    } //end of class Program
}

Environment & Configuration

• Windows 10
• .NET 8.0
• LLamaSharp version: 0.18
• CUDA version (if you are using cuda backend): CPU backend
• CPU & GPU device: x64

Known Workarounds

No response

[martindevans]

The null refererence exception is caused by the InferenceParams.SamplingPipeline being null. It's meant to default to the DefaultSamplingPipeline if you don't set it, so that looks like a bug in the LLamaInteractExecutor.

[robertmuehsig]

Adding this line should fix it in 0.18.0

InferenceParams inferenceParams = new InferenceParams()
{
    SamplingPipeline = new DefaultSamplingPipeline(), // Use default sampling pipeline. <-- not in sample
    MaxTokens = 256, // No more than 256 tokens should appear in answer. Remove it if antiprompt is enough for control.
    AntiPrompts = new List<string> { "User:" } // Stop generation once antiprompts appear.
};

[grammophone]

Thank you for your prompt responses, @martindevans and @robertmuehsig! Indeed, specifying the pipeline worked.