search

yipoh / AesBench

An expert benchmark aiming to comprehensively evaluate the aesthetic perception capacities of MLLMs.
Apache License 2.0
184 stars 5 forks source link

Evaluation code #2

Closed zxy110 closed 6 months ago

zxy110 commented 6 months ago

Hi，it seems the evaluation prompt of AesA and AesI is not consistent with paper.

The code version: [Question] Choose one from the following options: [Options] You should output the correct option without explanation.

The paper version: AesA: #User: How is the aesthetic quality of this image? [Image token]Choose one from the following options: A) High B) Medium C) Low. You should output the correct option without explanation. AesI: #User: This is an image with [high] aesthetics, please explain the reasons that are related to the aesthetic quality, where [high] is replaced by the actual aesthetic

dylanqyuan commented 6 months ago

Thanks for your attention! 😄 The question is: evaluation prompt of code version is not aligned with paper version.

The answer is: In AesBench, AesE, AesP, AesA related to a option, which aligned with the code version you mentioned: [Question] Choose one from the following options: [Options] You should output the correct option without explanation.

Here [Question]=How is the aesthetic quality of this image in the AesA example you listed. AesI examine whether MLLMs can accurately interpret the reasons for the aesthetic quality. So the answer of this task is NO option.(See Section 4.3) :page_facing_up:

Hope this response adequately addresses the concerns. Should there be any further questions or additional assistance required, please don't hesitate to ask. :speech_balloon: The contribution to the project is greatly appreciated. :heart:

zxy110 commented 6 months ago

Thank you very much for your response. All of my concerns have been addressed.