Error when substituting model in example

psschwei commented 1 week ago

Expected Behavior

Starter framework example works when swapping in another model other than llama3.1

Actual Behavior

Running the example returns an error: LinePrefixParserError: The generated output does not adhere to the schema. Parsed text line corresponds to a node "tool_name" which is not a start node!

Steps to Reproduce the Problem

Clone the repo and do the initial setup: https://github.com/i-am-bee/bee-agent-framework-starter/blob/aa4ab8d048ffede9efef1d59413dbf8eeabee44a/README.md?plain=1#L3-L7
Download the granite-code model
```
ollama pull granite-code
```

Update the modelId to use granite-code

diff --git a/src/agent.ts b/src/agent.ts
index 57b4c0d..0a9d8b2 100644
--- a/src/agent.ts
+++ b/src/agent.ts
@@ -9,7 +9,7 @@ import * as fs from "node:fs";
import * as process from "node:process";

const llm = new OllamaChatLLM({
-  modelId: "llama3.1",
+  modelId: "granite-code",
parameters: {
 temperature: 0,
 repeat_penalty: 1,

Run the example:
```
$ npm run start
```

start tsx ./src/agent.ts -p tsconfig.json

User 👤 : What is the current weather in Las Vegas? [AgentError: Error has occurred!] { isFatal: false, isRetryable: true, context: {},

[LLMError: LLM has occurred an error.] {
  isFatal: false,
  isRetryable: true,
  context: {},
  [errors]: [
    [LinePrefixParserError: The generated output does not adhere to the schema.
    Parsed text line corresponds to a node "tool_name" which is not a start node!] {
      isFatal: true,
      isRetryable: false,
      context: {
        lines: 'Function Name:',
        excludedLines: '',
        finalState: {},
        partialState: {}
      },
      [errors]: []
    }
  ]
}

] }

planetf1 commented 1 week ago

I hit a similar error (schema). However on checking I was still running an earlier version of the framework. Same results as Paul above, but also tried llama3.2 which fails with:

User 👤 : What is the current weather in London?
(node:50179) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
Agent 🤖 (thought) : The user wants to know the current weather in London. I can use the OpenMeteo function to retrieve the current weather forecast for London.
Agent 🤖 (tool_name) : OpenMeteo
Agent 🤖 (tool_input) : {"location": {"name": "London", "country": "United Kingdom", "language": "English"}}
Agent 🤖 (tool_caption) : Get current weather forecast for London
Agent 🤖 (tool_output) : ToolInputValidationError: The received tool input does not match the expected schema.
Input Schema: "{"type":"object","properties":{"location":{"anyOf":[{"type":"object","properties":{"name":{"type":"string","minLength":1},"country":{"type":"string"},"language":{"type":"string","default":"English"}},"required":["name"],"additionalProperties":false},{"type":"object","properties":{"latitude":{"type":"number"},"longitude":{"type":"number"}},"required":["latitude","longitude"],"additionalProperties":false}]},"start_date":{"type":"string","format":"date","description":"Start date for the weather forecast in the format YYYY-MM-DD (UTC)"},"end_date":{"type":"string","format":"date","description":"End date for the weather forecast in the format YYYY-MM-DD (UTC)"},"temperature_unit":{"type":"string","enum":["celsius","fahrenheit"],"default":"celsius"}},"required":["location","start_date"],"additionalProperties":false,"$schema":"http://json-schema.org/draft-07/schema#"}"
Validation Errors: [{"instancePath":"","schemaPath":"#/required","keyword":"required","params":{"missingProperty":"start_date"},"message":"must have required property 'start_date'"}]

HINT: If you're convinced that the input was correct but the function cannot process it then use a different function or say I don't know.
Agent 🤖 (final_answer) : I don't know how to provide the current weather in London using the OpenMeteo function. Can you provide more information about the location or the date you're interested in?
Agent 🤖 : I don't know how to provide the current weather in London using the OpenMeteo function. Can you provide more information about the location or the date you're interested in?

(Note - I'd changed city to London. That was working with llama3.1)

planetf1 commented 1 week ago

I note that in the failing case, the tool is provided with a location:

{"location": {"name": "London", "country": "United Kingdom", "language": "English"}}

whilst in my working case the tool is provided with

{"location": {"latitude": 51.5074, "longitude": -0.1278}, "start_date": "2024-10-09", "end_date": "2024-10-09", "temperature_unit": "celsius"}

This is quite different to the granite-code case as here it is about the results from the model, as opposed to a schema error actually getting a response from the model.

planetf1 commented 1 week ago

I also tried an ibm-granite model from hugging face, using the following ollama Modelfile:

git clone https://huggingface.co/ibm-granite/granite-8b-code-instruct-4k-GGUF

Then created an ollama model using the following file

 # Modelfile
 FROM "granite-8b-code-instruct-4k-GGUF/granite-8b-code-instruct.Q4_K_M.gguf"

by

 ollama create granite-8b-code-instruct-4k -f granite-8b-code-instruct-4k.Modelfile

With this model I got different output:

 User 👤 : What is the current weather in Las Vegas?
(node:85892) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
Agent 🤖 (thought) : The user wants to know the current weather in Las Vegas. I can use the OpenMeteo function to retrieve this information.
Agent 🤖 (tool_name) : OpenMeteo
Agent 🤖 (tool_input) : {"location":{"name":"Las Vegas","country":"USA","language":"English"},"start_date":"2024-10-09","end_date":"2024-10-09","temperature_unit":"celsius"}
Agent 🤖 (tool_caption) : Retrieve the current weather in Las Vegas.
Agent 🤖 (tool_output) : {"latitude":36.16438,"longitude":-115.14392,"generationtime_ms":0.12004375457763672,"utc_offset_seconds":0,"timezone":"UTC","timezone_abbreviation":"UTC","elevation":609,"current_units":{"time":"iso8601","interval":"seconds","temperature_2m":"°C","rain":"mm","apparent_temperature":"°C"},"current":{"time":"2024-10-09T11:30","interval":900,"temperature_2m":21.9,"rain":0,"apparent_temperature":17.8},"hourly_units":{"time":"iso8601","temperature_2m":"°C","relative_humidity_2m":"%","apparent_temperature":"°C"},"hourly":{"time":["2024-10-09T00:00","2024-10-09T01:00","2024-10-09T02:00","2024-10-09T03:00","2024-10-09T04:00","2024-10-09T05:00","2024-10-09T06:00","2024-10-09T07:00","2024-10-09T08:00","2024-10-09T09:00","2024-10-09T10:00","2024-10-09T11:00","2024-10-09T12:00","2024-10-09T13:00","2024-10-09T14:00","2024-10-09T15:00","2024-10-09T16:00","2024-10-09T17:00","2024-10-09T18:00","2024-10-09T19:00","2024-10-09T20:00","2024-10-09T21:00","2024-10-09T22:00","2024-10-09T23:00"],"temperature_2m":[36.5,34.3,32.6,31.8,29.1,28.6,26.7,26.8,25.7,24.6,23.1,22,22,21.8,21.3,23.8,27.8,30.1,32.5,34.1,35.1,36.3,36.9,37],"relative_humidity_2m":[8,8,8,6,12,11,13,12,14,14,16,17,16,16,18,19,16,16,14,11,9,8,7,6],"apparent_temperature":[32.4,30.4,28.9,28,24.7,24.5,23.7,23.5,21.7,20.1,19.5,18.3,17.8,17.7,16.9,20.6,25.3,27.1,30.1,32.1,33,33.6,32.9,31.7]},"daily_units":{"time":"iso8601","apparent_temperature_max":"°C","apparent_temperature_min":"°C","sunrise":"iso8601","sunset":"iso8601"},"daily":{"time":["2024-10-09"],"apparent_temperature_max":[33.6],"apparent_temperature_min":[16.9],"sunrise":["2024-10-09T13:43"],"sunset":["2024-10-10T01:12"]}}
[AgentError: Error has occurred!] {
  isFatal: false,
  isRetryable: true,
  context: {},
  [errors]: [
    [LinePrefixParserError: The generated output does not adhere to the schema.
    Nothing valid has been parsed yet!] {
      isFatal: true,
      isRetryable: false,
      context: {
        lines: '',
        excludedLines: '',
        finalState: {},
        partialState: {}
      },
      [errors]: []
    }
  ]
}

Tomas2D commented 5 days ago

Granite models are not supported right now. They don't follow our system prompts.

mmurad2 commented 5 days ago

Wanted to clarify that granite support is on our roadmap for Q4

Tomas2D commented 5 days ago

We will at least try to provide an example (maybe a tutorial) of how to build your own agent using a framework.

i-am-bee / bee-agent-framework-starter