search

gptscript-ai / desktop

MIT License
22 stars 13 forks source link

LLM is not able to respond to chat messages when knowledge tool call fails for an assistant with `workspace` tool and dynamically added knowledge tool. #264

Closed sangee2004 closed 1 month ago

sangee2004 commented 2 months ago

Electron build from bf86f1

Steps to reproduce the problem:

  1. Create an assistant with workspace tool.
  2. Chat with this assistant and add files using "Add Knowledge"
  3. Able to chat with the assistant and get information about knowledge files.
  4. Ask question about anything other than the knowledge file.
  5. Knowledge tool handles the query and fails to retrieve any info which is as expected. But the query is not handled by LLM after this failure and user gets no response for the messages.
Screenshot 2024-08-22 at 3 49 53 PM Screenshot 2024-08-22 at 3 49 47 PM

Note - This behavior is not seen when testing with Tildy assistant and adding files using "Add Knowledge" tool .

cjellick commented 2 months ago

@iwilltry42 i would expect this to be addressed by you reworking the knowledge tool into that context-based querier paradigm. assigning to you and using this issue as a place holder for that task

sangee2004 commented 2 months ago

@iwilltry42 When testing with latest version of desktop with commit id -d2d9d7ec and gptscript version - v0.0.0-dev+fbb8f5d4 , I am not seeing this issue reproduce.

  1. Create an assistant with workspace tool.
  2. Chat with this assistant and add files using "Add Knowledge"
  3. Able to chat with the assistant and get information about knowledge files.
  4. Ask question about anything other than the knowledge file.

I see this succeed . I see one of the 2 things happen for this query

  1. knowledge tool gets called , returns no match and it resorts to LLM providing the response Screenshot 2024-08-28 at 9 43 01 PM
  2. knowledge tool does not get called and LLM responds directly to this query. Screenshot 2024-08-28 at 9 45 15 PM
iwilltry42 commented 2 months ago

Large changes to how knowledge is integrated: #385

sangee2004 commented 1 month ago

Blocked on testing with workflow because of https://github.com/gptscript-ai/desktop/issues/425

iwilltry42 commented 1 month ago

Mitigations and debugging support landed here: https://github.com/gptscript-ai/knowledge/releases/tag/v0.4.13 Waiting for https://github.com/gptscript-ai/desktop/pull/450 to be merged

sangee2004 commented 1 month ago

Blocked on testing this workflow - https://github.com/gptscript-ai/desktop/issues/425

sangee2004 commented 1 month ago

Tested with a66cd29ecd This issue is not seen anymore.