Prompt dataset for testing performance of response

michaelthwan / searchGPT

Grounded search engine (i.e. with source reference) based on LLM / ChatGPT / OpenAI API. It supports web search, file content search etc.

MIT License

621 stars 65 forks source link

Prompt dataset for testing performance of response #15

Open michaelthwan opened 1 year ago

michaelthwan commented 1 year ago

We may have a gut feeling of how is the quality of response from current pipeline. But we may want a boarder range of prompts with different context.

For example

Normal question (What is ChatGPT?)
Strange question (What is the best way to eat metal ball?)

eren23 commented 1 year ago

Hey, great idea! :) Will this dataset be manually created pairs of questions and answers for normal/strange questions to compare against the pipeline?

michaelthwan commented 1 year ago

@eren23 I think some huggingface dataset may contain different kinds of prompts and thus no need to self-build. But in fact, I have more than enough prompts by users' input in the demo page 😂