Creating a new benchmark answers

Gautam-Rajeev commented 3 months ago

We need to have golden answers for all these questions populated here

(some benchmark answers already exist for 350 questions) from the content and the previous answers. @aashutosh-samagra can pick from there and do vlookup

For each question, can we use the content to create an answer using GPT , we can use whatever is coming from the Bot if it looks like a good answer, if no relevant answer is there on content or question is not clear, the answer should be -'Sorry I cannot answer the query because we dont have content for it' or 'Can you clarify the question'

Gautam-Rajeev commented 3 months ago

The sheet that needs to be populated is here

Steps to figure out the issues :

I have created a column called 'Issues'. You can fill them up in comma separated values. This may not be the breaking change that caused the problem/mistake. This is just an issue in the bot which you see for the question that needs to be fixed. Some questions may have multiple problems. Some questions may have correct answer and yet have problems. e.g. There was no content but the answer is correct (very common) because GPT took over. or question was classified wrong but still has no issues.

The possible issues are:

translate : Translate translated wrongly or partially. e.g : मोटे अनाज में मंडवा का क्या महत्व है got translated to '"What is the importance of mandwa in coarse grains"' when it should have been '"What is the importance of finger millet in coarse grains"'. If the translate is wrong, write the corrected translate in the column 'Correct Translate'
Classifier: The classifier classified the question incorrectly. e.g. "What is nalwa list" should be non_agri not 'pest/scheme'
Content: There is no relevant content retrieved in the Top20Chunks column. You can usually verify by doing a simple 'Ctrl+F' search for keywords required to answer . Warning there is going to be repeated content between Summary and Content because we have copied the Content into Summary for older content where we didn't summarize.
Retrieval : There is no relevant content in 'ContentPicked' but its there in Top20Chunks.
GPT_non_agri : This happens when the bot responds that the question is not agri related when the question is agri related. P.S. This is not a 'Classifier' error. It could also be a Classifier error if the classifier responds something other than non_agri. But if bot responds not related to agriculture, then its a 'GPT_non_agri' error.
GPT_error: The relevant content was there but GPT made some mistake in answering or phrasing or picking important info

Fill the column 'Correct Translate' and 'Correct Class' if there was a mistake in classification/translate.
Fill column called 'Intents' : The column covers what the farmer was trying to ask in the question. It should ideally have terms like pest prevention, pest treatment, disease treatment, disease prevention, scheme information, scheme eligibility, scheme status, fertilizer application, irrigation, agriculture best practices etc. You are free to write anything you find correct. We will cluster later. But keep the intent compact and comma separate different intent.
Fill the column Correct/Wrong in answers. If the answer is bad only because of a lack of content and gives a generally correct answer its filled as Correct. If it gives an answer which is not useful to the user at all or is irrelevant, its marked as Wrong
For the cases where the content was there or the question was not related to agriculture fill the golden answer, constructing an answer from the question and content. For where the answer went wrong because of the classification/translate. Use the admin panel to look at relevant content and fill the golden answer from the content. One can use GPT and ask it to answer basis of question and answer in English.

For cases where you have content as an issues, do not give an answer. You can add source of content if you feel like you have time or its useful,