search

logancyang / obsidian-copilot

A ChatGPT Copilot in Obsidian
https://www.obsidiancopilot.com/
GNU Affero General Public License v3.0
2.34k stars 158 forks source link

Adding Claude API breaks QA modes #355

Open FamedBear16 opened 4 months ago

FamedBear16 commented 4 months ago

Describe the bug Today I added the Claude 3 API key. The QA single note and QA Vault (with both Claude 3 and Chat GPT) stopped giving me answers related to my notes.

When I delete the Claude API key, everything works as expected.

To Reproduce I just added the Claude API and it stopped working well. I am using Chat GPT Api

petery789 commented 4 months ago

Hey @FamedBear16! Looks like it's working fine for me. The steps that I followed:

  1. reset to default
  2. add openai key
  3. play with Long Note QA and it works
  4. add claude key
  5. play with Long Note QA and it works

Although I haven't been able to get the ideal response from vault QA, single note seems to work fine for me. Would you be more specific on reproducing steps?

Version: 2.5.1

FamedBear16 commented 4 months ago

Hi @petery789,

I reset to defaults, changed to Chat GPT 16K, use 7 notes max for the Vault I did several tests. Now it seems that Open AI is working fine (after several trial on closing and opening Obsidian). However, Claude AI model is not using the QA context, it is just giving me a generic answer as if I was chatting. He also replies in French. Here are a few screenshots.

image

image

image

image

FamedBear16 commented 4 months ago

Hi @petery789 This is an example of how Claude AI is not using the QA mode, but it goes in chat mode, even if I select QA. I tested it for both vault and single note and it does not work. The second is the answer from GPT

image

image

logancyang commented 4 months ago

Can you turn on debug mode and include the console in your screenshots?

To debug QA modes, you need to make sure your embedding provider is set, and the indexing process finished correctly. Check my demo video, especially the troubleshoot part for instruction https://youtu.be/NSoKRYNlOls?si=jZQYpv33NSjcMe2B.

Personally, my experience with Claude in QA modes is much better than GPT-3.5 16K, closer to GPT-4-TURBO.

As for why Claude replied in French, does your note have French? Or your system prompt in settings? Otherwise I see no reason.🤔

FamedBear16 commented 4 months ago

@logancyang, This is my test: I am using Open AI Embedding small. GP4-Turbo works well. Claude 3 does not work at ll in QA for a single note.

image

image

with GP4-TURBO works well image

FamedBear16 commented 4 months ago

@logancyang This is for thh entire vault. Claude 3 does not work. GPT 3.5 works but is much worse than single Note QA.

YOu can see that it replies in Russin, I had it also in Spanish and French. I do no use this languaes in the vault image

image

image

FamedBear16 commented 4 months ago

I have noticed that sometimes Claude 3 manage to give an answer. I believe that there is something going on.

FamedBear16 commented 4 months ago

This is another test. It shows you how Clause 3 fails to enter QA mode and use Chat mode instead image

image

image image

logancyang commented 4 months ago

From your last screenshot, it seems Claude failed to connect. Can you try refreshing your app with the console open, switching from and to Claude several times. Personally I haven't seen Claude connection issue. Do you use a VPN or something?

As for the different languages, it's a known issue with old gpt 3.5 16K. Please use GPT 3.5, it's the new one and already have 16K context window, the GPT 3.5 16K is a legacy model. For best results, gpt 4 turbo or Claude 3 are recommended.

FamedBear16 commented 4 months ago

@logancyang,

I thought that the last one was GPT 3.5 16K, I will move to GPT 3.5. I have the language issue with Claude 3. In addition, the latter when it fails to connect, I just get an error message, so this is not a problem, I just give it another go.

So far it seems that in QA mode, Claude 3, sometiems says that the context provided does not have an answer (but GPT works well), other times it just works as it was in Chat mode, and more rarely it works well.

FamedBear16 commented 4 months ago

@logancyang,

This is another screenshot when I sue Claude 3 in QA. |it gives me a response, but it does not seem to use my note. GPT 4 works fine. I am not using a VPN.

image

logancyang commented 4 months ago

@FamedBear16 The part I don't understand is that why you get CORS error from OpenAI embedding API? Haven't seen that before. You also get connection error to your local proxy 53001. Looks like a firewall thing? 🤔

FamedBear16 commented 4 months ago

@logancyang I don't have any idea of what these errors mean. The CORS does not happen often, I managed to Embedded my entire vault. Open AI is working well for me. I hope that with futures updates I will also be able to use Claude 3 in QA mode. The chat mode is working well. I am happy to do some debugging in case.

FamedBear16 commented 4 months ago

@logancyang, I tried again "single note QA" with Claude 3, and it completely disregards the Note context. GPT-4 Turbo on the contrary works well. The note was about ACT Therapy, while Claude replied with a generic answer based on DBT

image