lz1oceani / verify_cot

Creative Commons Attribution 4.0 International
103 stars 8 forks source link

Add gt_answer to last_letter and date. #2

Closed AegeanYan closed 9 months ago

AegeanYan commented 9 months ago

Dear authors, I want to make use of your data but the gt_answer in date and last_letter seems not available. While I think you also changed the question format, so I cannot automatically annotate by myself with date of BIG BENCH, could you kindly update it. Thank you.

lz1oceani commented 9 months ago

We use the following code to reformat the date dataset. _ = example of date noisytail = " in MM/DD/YYYY?" assert ["question"].endswith(noisytail), "Question should end with 'in MM/DD/YYYY?'" ["question"] = _["question"].replace(noisy_tail, "? Please answer in format MM/DD/YYYY.")

For last letters, we use dataset here: https://github.com/kojima-takeshi188/zero_shot_cot/blob/main/dataset/last_letters/last_letters.json .