Include VCR

[tianyu-z]

• A descriptive title: VCR: Vision_Caption_Restoration
• A detailed description We introduce Visual Caption Restoration (VCR), a novel vision-language task that challenges models to accurately restore partially obscured texts using pixel-level hints within images. This task stems from the observation that text embedded in images is intrinsically different from common visual elements and natural language due to the need to align the modalities of vision, text, and text embedded in images. While numerous works have integrated text embedded in images into visual question-answering tasks, approaches to these tasks generally rely on optical character recognition or masked language modeling, thus reducing the task to mainly text-based processing. However, text-based processing becomes ineffective in VCR as accurate text restoration depends on the combined information from provided images, context, and subtle cues from the tiny exposed areas of masked texts. We develop a pipeline to generate synthetic images for the VCR task using image-caption pairs, with adjustable caption visibility to control the task difficulty. With this pipeline, we construct a dataset for VCR called VCR-Wiki using images with captions from Wikipedia, comprising 2.11M English and 346K Chinese entities in both easy and hard split variants.

• VCR-Wiki (vcr_wiki)
  • VCR English easy mode
  • VCR English easy mode full test set (vcr_wiki_en_easy)
  • VCR English easy mode test 100 (vcr_wiki_en_easy_100)
  • VCR English easy mode test 500 (vcr_wiki_en_easy_500)
  • VCR English hard mode
  • VCR English hard mode full test set (vcr_wiki_en_hard)
  • VCR English hard mode test 100 (vcr_wiki_en_hard_100)
  • VCR English hard mode test 500 (vcr_wiki_en_hard_500)
  • VCR Chinese easy mode
  • VCR Chinese easy mode full test set (vcr_wiki_zh_easy)
  • VCR Chinese easy mode test 100 (vcr_wiki_zh_easy_100)
  • VCR Chinese easy mode test 500 (vcr_wiki_zh_easy_500)
  • VCR Chinese hard mode
  • VCR Chinese hard mode full test set (vcr_wiki_zh_hard)
  • VCR Chinese hard mode test 100 (vcr_wiki_zh_hard_100)
  • VCR Chinese hard mode test 500 (vcr_wiki_zh_hard_500)

[Luodian]

Thanks for this PR! (and thanks for resolving previously naming issues)

Could you please do one more task by pasting an evaluation results with this PR?

[sheryc]

Below's a screenshot for the task. We test one of our subtasks (vcr_wiki_en_easy) with openbmb/MiniCPM-V. The output contains jaccard and exact_match scores, while our code also generates json files in the log folder.

[Luodian]

That looks great, but a new conflict emerges since we merged another PR before this. Would you mind solving the conflict by directly accept the incoming changes on README.md

Then we can directly merge your PR.

Thanks!

[sheryc]

The upstream changes are merged. Could you please check if our PR is ready for merging? Thanks!