Question about BLIPvqa code with reproduce scores of shape, texture

Karine-Huang / T2I-CompBench

[Neurips 2023] T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation

https://arxiv.org/pdf/2307.06350.pdf

MIT License

168 stars 5 forks source link

Question about BLIPvqa code with reproduce scores of shape, texture #8

Closed ofzlo closed 9 months ago

ofzlo commented 9 months ago

Hi, thank you for sharing your excellent work.

I'm interested in reproducing the B-VQA scores presented in your paper, specifically those in tables 2, 3, and 4. While reviewing the /BLIPvqa_eval/BLIP_vqa.py code, I noticed that it primarily uses 'color' as a parameter. I'd like to inquire whether it can be adapted to work with 'shape' and 'texture' as well. Additionally, I'm curious if making this change would have any impact on the score production.

Retr0573 commented 4 months ago

I'm curious about it too, do you have any understanding now?

ofzlo commented 4 months ago

@Retr0573 I apologize for not being able to provide a more detailed explanation, as it was quite some time ago, but I believe it was not an issue because 'color', 'shape', and 'texture' were using the same JSON file.