Responses include chain of thought intermediate steps

[maxyousif15]

I have noticed that the output of the LLMRails object occasionally includes content which seemingly is related to predicted next question by the user, or chain of thought outcomes. Below is a screenshot to highlight this issue. Is there any guidance on this?

I expect the LLM to respond to the question, without including all the other additional content. It's extremely hard to debug what is going on.

I've also included my config.yml and an example rails.co file below.

config.yml

models:
  - type: main
    engine: openai
    model: gpt-3.5-turbo-1106
    parameters:
      temperature: 1

  - type: embeddings
    engine: SentenceTransformers
    model: BAAI/bge-base-en-v1.5

streaming: True

instructions:
  - type: general
    content: |
      You are a helpful phone expert assistant.
      Your job is to answer questions about product specifications.

rails.co

define user ask politics
    "what are your political beliefs?"
    "thoughts on the president?"
    "thoughts on the prime minister?"
    "thoughts on a political party?"
    "dictator"
    "manifesto"
    "political mandate"
    "Sadiq Khan"
    "left wing"
    "right wing"

define bot answer politics
    "I'm a shopping assistant, I cannot speak on politics. Is there anything else I can help you with?"

define flow politics
    user ask politics
    bot answer politics

define user ask write code
    "create a script"
    "could you write me some code?"
    "show me some code to do something"
    "write some code in python"
    "write some code in javascript"
    "write some code in c++"
    "write some code in typescript"
    "write some code in r"
    "write some code in rust"
    "write some code in sql"

define bot ask write code
    "I'm a shopping assistant, I cannot write code. Is there anything else I can help you with?"

define flow write code
    user ask write code
    bot answer write code

define user ask about competitor
    "alternative companies"
    "why are you more expensive"
    "why do you offer less services for more money"

define bot answer about competitor
    "I cannot discuss competitor options. Is there anything else I can help you with?"
    "I cannot answer queries on competitor options. You will have to navigate to the respective websites for answers to your queries. Is there anything else I can help you with?"

define flow competitor
    user ask about competitor
    bot answer about competitor

Any insights would be much appreciated.

[drazvan]

@maxyousif15 : it looks like the LLM is not stopping correctly. @trebedea can you try to reproduce?

[trebedea]

Hi @maxyousif15 ! I have not been able to replicate this bug. I have run with the configs you provided and the streaming example demo that I also assume you are also using and I always receive a normal bot message. I've run 50 times, just to ensure that it's not something that arises from time to time, mainly due to the high temperature (I've seen you use temperature=1 for bot message generation).

Can you share additional details to be able to replicate? Is the problem still there?