Cannot reproduce the result (low accuracy for counting evaluation)

[YuhaoChrisJ]

Hi, thanks for the great work. However, I have problems when reproducing the counting accuracy result.

• I used the provided layout file "gpt4.counting.k-similar.k_8.px_64.json" under "llm_output/counting/" to generate images.
• Using the provided yml file, I have torch vision conflict problem when running evaluation code. So I used docker environment provided by GLIP and got the code to run successfully.
• However, for counting accuracy, the evaluation result is only 23%. I checked the output of the images by GLIP, the detecting result seems unsatisfying. However, the layout accuracy evaluation performs well, so I assumed that's not GLIP's problem.

Any hint for what might have caused the problem is helpful!

[weixi-feng]

Hi,

Sorry for the late response. I found that there were some bugs in the code of computing accuracy for object counting. I have updated the script eval_models/GLIP/eval_counting.py and you can also get my GLIP eval results here. Unzip the file into eval_models/GLIP/ and run the script to get the accuracy immediately.

I didn't check all the outputs by GLIP but I do notice many cases where the grounding results are far from satisfying. I think that's an open problem with detection/grounding models on generated images.

Let me know if it helps and if there are any remaining issues.