search

pashpashpash / vault-ai

OP Vault ChatGPT: Give ChatGPT long-term memory using the OP Stack (OpenAI + Pinecone Vector Database). Upload your own custom knowledge base files (PDF, txt, epub, etc) using a simple React frontend.
https://vault.pash.city
MIT License
3.27k stars 308 forks source link

Context Does Not Provide Information? #26

Open gitardav opened 1 year ago

gitardav commented 1 year ago

Hello,

How might I improve response quality's reasoning? Is there a setting, prompt technique, or limitation I need to know more about? Does this solution lack some sort of access to ChatGPT's training data as context and reasoning capability?

I've attached screen shots. There seems to be a notable difference in response quality when I compare this solution's responses with ChatGPT responses.

Whereas ChatGPT has capacity to reason and improvise within its responses, this solution's responses seem more like web search results.

When I provide the same content as reference and use the same prompt, ChatGPT offers what I consider to be a much higher quality response. ChatGPT understood what 'urgent vs important' framework meant, without my assistance, and applied that framework to the reference material.

By contrast, this solution seems to be focused on where literal relevance in uploaded documents might be found.

Thank you for sharing this solution with the community. I am encouraged by its potential.

vaultop-response chatgpt-response
WizKid1968 commented 1 year ago

You have to upload a text file and direct your questions towards the file you uploaded. This isn't a version of chatgpt.

pashpashpash commented 1 year ago

This is by design – I have the temperature parameter set close to zero, meaning the AI will not "improvise" outside of the context it is provided with. The primary goal is to only answer based on the context in the documents you upload.

What behavior would you like to see? Perhaps there can be a feature made out of what you desire

gitardav commented 1 year ago

Thank you for explaining. Let me know if you would like to discuss this and other use cases.

I can imagine two behaviors:

  1. I would like to be able to synthesize two documents. Imagine this scenario: Document A’s content describes a method for organizing topics (for example, the Urgent vs Important quad framework for prioritizing things). Document B contains content somewhat relevant to Document A’s categories and structure but Document B’s content is organized per its own structure or context. However, I’d like the app’s responses to quickly categorize or organize Document B’s most relevant content per Document A’s described way of organizing things.

  2. If possible, I would like to be able to configure the temperature setting for my instance.

Thank you for your time