search

microsoft / chat-copilot

MIT License
2.02k stars 688 forks source link

Can't retrieve document memory in chat #979

Closed glenq closed 3 months ago

glenq commented 4 months ago
  1. I've initiated a chat session and uploaded a Word document.
  2. My aim is to utilize AI to rewrite the document with a specified Markdown style.

However, the Copilot indicates it can't access the document. The document should already be stored in the "DocumentMemory" and should be accessible for the conversion.

Are there any errors here? Or is there a misunderstanding on my part? What do I need to do in order to read the document content during the session?

image

ahmadabuhattab commented 4 months ago

I am looking into this.

glenq commented 4 months ago

I am looking into this.

@ahmadabuhattab I discovered that this issue was cuased by a previous bug fix. After changing the minRelevance param to 0 instead of relevanceThreshold, then it's now working correctly. The document memory can now be retrieved.

Here is the commit with the bug fix code: https://github.com/microsoft/chat-copilot/commit/b0fe77ae2f9b71e5c2b529e5bd7288e3b0123c05

glenq commented 4 months ago

@ahmadabuhattab More: The default value is 0.8: internal float DocumentMemoryMinRelevance { get; } = 0.8F; However, I've tested many documents, and all have a relevance below 0.75. Any suggest?

Test result:

image

ahmadabuhattab commented 4 months ago

I am looking into this.

@ahmadabuhattab I discovered that this issue was cuased by a previous bug fix. After changing the minRelevance param to 0 instead of relevanceThreshold, then it's now working correctly. The document memory can now be retrieved.

Here is the commit with the bug fix code: b0fe77a

That's amazing, it failed some checks though, but I see how you did it. Let me know if you want me to fix them.

ahmadabuhattab commented 4 months ago

The relevance score threshold in the system is set at 0.8, which is higher than the scores your documents are receiving (should be a small issue with the memory thing). Try adjusting the DocumentMemoryMinRelevance parameter if that is available. This will help to lower the value to 0.7....and to see if that allows more documents to pass the relevance check. Let me know if you need me to step in.

glahaye commented 3 months ago

For good measure, the default DocumentMemoryMinRelevance value was lowered from 0.8 to 0.66 in #1006