LLMAgent doesn't support Ollama (needs LITS update)

[joshuacox]

How do I prevent openAI from being invoked at all?

[LlamaIndex] OpenAIError: The OPENAI_API_KEY environment variable is missing or empty; either provide it, or instantiate the OpenAI client with an apiKey option, like new OpenAI({ apiKey: 'My API Key' }).
    at new OpenAI (webpack-internal:///(rsc)/./node_modules/openai/index.mjs:69:19)
    at eval (webpack-internal:///(rsc)/./node_modules/@llamaindex/openai/dist/index.js:772:28)
    at async OpenAI.streamChat (webpack-internal:///(rsc)/./node_modules/@llamaindex/openai/dist/index.js:888:31)
    at async asyncGeneratorWrapper (webpack-internal:///(rsc)/./node_modules/@llamaindex/core/decorator/dist/index.js:32:45)
    at async response.<computed> (webpack-internal:///(rsc)/./node_modules/@llamaindex/core/decorator/dist/index.js:99:34)
    at async Object.start (webpack-internal:///(rsc)/./node_modules/@llamaindex/core/agent/dist/index.js:35:30)
 POST /api/chat 500 in 3533ms
The data stream is hanging. Did you

.env

MODEL_PROVIDER=ollama
MODEL=llama3.2
EMBEDDING_MODEL="sentence-transformers/all-mpnet-base-v2"
EMBEDDING_DIM=1024
FILESERVER_URL_PREFIX=http://localhost:3000/api/files
NEXT_QUESTION_PROMPT="You're a helpful assistant! Your task is to suggest the next question that user might ask. 
Here is the conversation history
---------------------
{conversation}
---------------------
Given the conversation history, please give me 3 questions that you might ask next!
Your answer should be wrapped in three sticks which follows the following format:

<question 1> <question 2> <question 3>

SYSTEM_PROMPT="You are a DuckDuckGo search agent. 
You can use the duckduckgo search tool to get information from the web to answer user questions.
For better results, you can specify the region parameter to get results from a specific region but it's optional.
"

[marcusschiesser]

@joshuacox we will upgrade llamaindex to fix this soon

[joshuacox]

I deleted all initializers aside from Ollama in settings.ts after taking this lines advice:

  // HINT: you can delete the initialization code for unused model providers

Which I did to no effect. It still complains about the missing API key. I tried setting a false key to get it to ignore the issue, but of course openAI rejects the API call. But I do not want it contacting openAI at all.

EDIT: of course I submitted that comment and then the refresh happened and I see the comment from you @marcusschiesser

[mcavdar]

hi @joshuacox Can you please initialize with npx create-llama@latest --pro rather than npx create-llama@latest, and update .env? This works for me.

[marcusschiesser]

should be fixed in Release 0.3.10 - please have a try

[mcavdar]

hi @marcusschiesser

Unfortunately, updating LlamaIndex to version 0.8.2 didn't resolve the issue. If it would be helpful, I can share the differences between the files when initialized with --pro versus without it.

without-pro/app/api/chat/engine:

import {
  BaseChatEngine,
  BaseToolWithCall,
  OpenAIAgent,
  QueryEngineTool,
} from "llamaindex";
import fs from "node:fs/promises";
import path from "node:path";
import { getDataSource } from "./index";
import { generateFilters } from "./queryFilter";
import { createTools } from "./tools";

export async function createChatEngine(documentIds?: string[], params?: any) {
  const tools: BaseToolWithCall[] = [];

  // Add a query engine tool if we have a data source
  // Delete this code if you don't have a data source
  const index = await getDataSource(params);
  if (index) {
    tools.push(
      new QueryEngineTool({
        queryEngine: index.asQueryEngine({
          preFilters: generateFilters(documentIds || []),
        }),
        metadata: {
          name: "data_query_engine",
          description: `A query engine for documents from your data source.`,
        },
      }),
    );
  }

  const configFile = path.join("config", "tools.json");
  let toolConfig: any;
  try {
    // add tools from config file if it exists
    toolConfig = JSON.parse(await fs.readFile(configFile, "utf8"));
  } catch (e) {
    console.info(`Could not read ${configFile} file. Using no tools.`);
  }
  if (toolConfig) {
    tools.push(...(await createTools(toolConfig)));
  }

  const agent = new OpenAIAgent({
    tools,
    systemPrompt: process.env.SYSTEM_PROMPT,
  }) as unknown as BaseChatEngine;

  return agent;
}

with-pro//app/api/chat/engine:

import { ContextChatEngine, Settings } from "llamaindex";
import { getDataSource } from "./index";
import { nodeCitationProcessor } from "./nodePostprocessors";
import { generateFilters } from "./queryFilter";

export async function createChatEngine(documentIds?: string[], params?: any) {
  const index = await getDataSource(params);
  if (!index) {
    throw new Error(
      `StorageContext is empty - call 'npm run generate' to generate the storage first`,
    );
  }
  const retriever = index.asRetriever({
    similarityTopK: process.env.TOP_K ? parseInt(process.env.TOP_K) : undefined,
    filters: generateFilters(documentIds || []),
  });

  const systemPrompt = process.env.SYSTEM_PROMPT;
  const citationPrompt = process.env.SYSTEM_CITATION_PROMPT;
  const prompt =
    [systemPrompt, citationPrompt].filter((p) => p).join("\n") || undefined;
  const nodePostprocessors = citationPrompt
    ? [nodeCitationProcessor]
    : undefined;

  return new ContextChatEngine({
    chatModel: Settings.llm,
    retriever,
    systemPrompt: prompt,
    nodePostprocessors,
  });
}

const agent = new OpenAIAgentI guess we need to fix this part.

[marcusschiesser]

@mcavdar great catch, it should be actually generate to use the LLMAgent instead of the OpenAIAgent - I fixed this in https://github.com/run-llama/create-llama/pull/410

[marcusschiesser]

Unfortunately, the Ollama LLM class from LlamaIndexTS doesn't support tool calls yet, I added this in https://github.com/run-llama/LlamaIndexTS/issues/1430

Once, this is added to LITS, it will automatically work in create-llama (after update). I keep this ticket open for tracking

[joshuacox]

sorry for the delay, but this is working great for me now, closing. Feel free to re-open if there are some other issues.

[marcusschiesser]

just for others, you need to update LllamaIndexTS if it doesn't work