How to employ ChatGPT as the evaluator in Section 3.3?

alibaba / GraphTranslator

GraphTranslator:Aligning Graph Model to Large Language Model for Open-ended Tasks

BSD 3-Clause "New" or "Revised" License

68 stars 12 forks source link

How to employ ChatGPT as the evaluator in Section 3.3? #10

Closed LITONG99 closed 2 months ago

LITONG99 commented 3 months ago

Dear authors, could you provide the template to employ ChatGPT as the evaluator to perform quantitative analysis?

According to the paper, the model is input with question-answering pairs of each test sample and uses the four-level (A, B, C, D) rating system. I am wondering what information is provided to the evaluator to make the judgment. For example, does it include the original text, neighborhood information, or the trained prompt?

Thank you.

fs302 commented 2 months ago

The evaluation prompt follows the reference paperSelf-instruct: Aligning language model with self generated instructions.. We provide user and neighbors information as background text, then ask ChatGPT to rate 3 question-answer pairs.