search

1jsingh / Divide-Evaluate-and-Refine

Repo for our NeurIPS 2023 paper on: Divide, Evaluate, and Refine: Evaluating and Improving Text-to-Image Alignment with Iterative VQA Feedback
https://1jsingh.github.io/divide-evaluate-and-refine
MIT License
24 stars 1 forks source link

Dataset with human data #1

Open vishaal27 opened 1 year ago

vishaal27 commented 1 year ago

Hey,

Thanks for the great work: I enjoyed reading the paper, and your proposed iterative improving strategy seems super interesting.

I am interested in the human data you collected (DecomposedCaptions4k with 24960 human annotations), is there a planned release date for this? Also a couple more related questions: 1) I could not find the supplementary material of your paper online (it wasn't attached to the arxiv paper), are you planning an update to arxiv; I would be super interested in going over the supp. mat. 2) For the human data, did you run two separate experiments? One for the preference study with two images and one prompt displayed, and another for single image-single prompt alignment?

Thanks!

1jsingh commented 1 year ago

Hi,

Thanks for your interest and positive comments about our paper. We plan to release the code and Decomposed-Captions-4k data together after refactoring to allow for easy use in downstream benchmarks. We currently plan to release the same by end August / early September.

Also regarding the other questions:

  1. I could not find the supplementary material of your paper online (it wasn't attached to the arxiv paper), are you planning an update to arxiv; I would be super interested in going over the supp. mat.

Thanks for pointing that! We will update the arxiv version to also include the supp. material.

  1. For the human data, did you run two separate experiments? One for the preference study with two images and one prompt displayed, and another for single image-single prompt alignment?

Yes, pairwise comparison scores are reported using a separate user-study, wherein subjects are shown a pair of images and asked to select the one with better alignment.

vishaal27 commented 1 year ago

Great, looking forward to the code release!

FlyFOV commented 1 year ago

Nice solution to alignment evaluation. Just wander wether the code/dataset will be released this Month~

vishaal27 commented 9 months ago

Hi @1jsingh, just checking to see if you had any updates about releasing the human annotations you collected?