search

pashpashpash / vault-ai

OP Vault ChatGPT: Give ChatGPT long-term memory using the OP Stack (OpenAI + Pinecone Vector Database). Upload your own custom knowledge base files (PDF, txt, epub, etc) using a simple React frontend.
https://vault.pash.city
MIT License
3.25k stars 306 forks source link

Responses appear to contain erroneous and inconsistent counts #35

Closed gitardav closed 1 year ago

gitardav commented 1 year ago

Issue: When prompting OP Vault to count the number of times a given word appears in the uploaded 1.7 MB XML data file, OP vault returns an erroneous and inconsistent count.

Steps to reproduce:

  1. Upload a document.
  2. Prompt OP Vault: "How many times does the word interoperability appear?" 2.1 OP Vault responds: The word "interoperability" appears 6 times in the given context
  3. Prompt OP Vault: "Count the number of times the word interoperability appears" 3.1 OP Vault responds: The word "interoperability" appears 9 times in the given context

In fact, the word interoperability appears 281 times in the given context.

pashpashpash commented 1 year ago

Your prompt will only count what is surfaced from the context (closest embeddings in pinecone db), not the entire document. This use case is not supported.