How are target_queries generated?

[c0ding4ever]

Sorry if I missed this in the repo somewhere, but could you let me know if there is a script to generate the JSON files in target_queries were generated (along with the correct formatting)?

[sleeepeer]

Hi, Thank you! In Section 5.1 of our paper, we discussed "Target questions and answers". We first used the ground truth context (qrel) to generate a correct answer for each query in the dataset using GPT-4, and then since we have already got the correct answer, we could use GPT-4 again to randomly generate an incorrect answer for each query. This is how we made the target_queries.

The script is a python file called gen_adv.py, we are modifying it and will upload later in this week. You could also email me if you have an urgent need for this file.

Thanks again for pointing it out!

[c0ding4ever]

Thanks for clarifying! I also wanted to ask if you generated incorrect answers via GPT-4 manually, and whether this was in a batch or independently, as I noticed that there are 100 target questions and answers for the QA datasets used in the paper.

Another question: will the code for generating adv_targeted_results/ be included in gen_adv.py as well? :)

[sleeepeer]

Hello, I just uploaded the script for generating adv_targeted_results in the latest commit. We found that target_queries and adv_targeted_results are redundant so we removed target_queries and just keep adv_targeted_results now.

And yes, as we discussed in Section 5.1, we mannually checked the quality of the generated incorrect answers via GPT-4. Now you could refer to gen_adv.py for more details.

[sleeepeer]

Hi @tamtakapanadze, I've fixed the path bug in the latest commit. Thanks for pointing it out!