Make sure stampy answers are given priority in responses

StampyAI / stampy-chat

Conversational chatbot to answer questions about AI Safety & Alignment based on information retrieved from the Alignment Research Dataset

https://chat.stampy.ai

MIT License

13 stars 7 forks source link

Make sure stampy answers are given priority in responses #64

Open Mishaall opened 1 year ago

FraserLee commented 1 year ago

Good issue, I agree. Does "given priority" mean we're boosting their ranking in semantic search results, or just re-shaping the prompt when they're present?

ccstan99 commented 1 year ago

The semantic search shouldn't be changed. But if a stampy question is among the results and similarity score is above a certain threshold (0.5?) the prompt should be to either very closely summarize or directly present the human-written stampy answer.

ccstan99 commented 1 year ago

This is actually a duplicate of issue #11 so I'll copy some comments below for reference and close that issue since this will have more complete information:

Up to date Stampy answers should be prioritized in responses with as little modification as possible.

From McGill Team's plan

Privilege curated sources in search: add biases to good/bad sources to make them more/less easily found by semantic search. Random personal blogs on lesswrong---, stampy+++, highly upvoted stuff ++, etc. Extending the length of good vectors by 0.1, and decreasing the length of bad vectors by 0.1, might work for this while still permitting search for ‘bad’ sources.

markovial commented 1 year ago

The semantic search shouldn't be changed. But if a stampy question is among the results and similarity score is above a certain threshold (0.5?) the prompt should be to either very closely summarize or directly present the human-written stampy answer.

I think we should just directly present the stampy answer, as it saves us a whole bunch of tokens. I'm not creating a new issue for prioritizing and directly displaying existing semantically similar answers because this issue seems to pretty much cover it.

ccstan99 commented 1 year ago

You can reference the code from followups.py that calls nlp.stampy.ai to find the nearest Stampy answer. Couple issues to be mindful:

If it's displayed as is, make sure the coloring or something clearly indicates it's a human-authored answer not generated including the original question title. Something prefix with "Here's a curated article that might answer your question..."
The actual score threshold to use might need some experimentation. 0.5 will likely find an FAQ that's highly relevant for the answer but might not be an "exact match"
Use the related questions as suggested followups afterwards