search

StampyAI / stampy-chat

Conversational chatbot to answer questions about AI Safety & Alignment based on information retrieved from the Alignment Research Dataset
https://chat.stampy.ai
MIT License
12 stars 5 forks source link

Log all queries in MySQL #83

Closed ccstan99 closed 9 months ago

ccstan99 commented 10 months ago

Log all queries and generated answers run (on deployed server) to MySQL. Might as well include sources cited also. Maybe also include whether called from GUI or simplified API or anything else which will can help isolate and track potential abuses?

Some columns: query: string with user query chunks: list of pinecone ids of chunks history: string or json/dict prompt: string with full prompt generated_answer: string moderation: json/dict results of moderation if flagged, or all moderation even if not flagged? thumbs_up: bool thumbs_down: bool comments: string for user to enter why thumbs down/up

FraserLee commented 10 months ago

The current webhook approach is pretty severely limited in bandwidth. If we were to do this, loads of other fun things would become possible even beyond the obvious security benefits (dumping and logging complete prompts and history for replication, more automated and faster prompt eng work, unblocking on the "thumbs up / thumbs down / flag for manual review" system, etc).

ccstan99 commented 10 months ago

Yes, my thoughts exactly!