Multimodal pre-trained models are trained on massive multimodal data, and they can utilize information from different modalities and perform various cross-modal tasks.
In this repository, we introduce VLE (Vision-Language Encoder), an image-text multimodal understanding model built on the pre-trained text and image encoders. It can be used for multimodal discriminative tasks such as visual question answering and image-text retrieval. Especially on the visual commonsense reasoning (VCR) task, which requires high-level language understanding and reasoning skills, VLE achieves the best performance among the public methods.
Recently, LLMs (Large Language Models) have achieved great success and have been used for a wide range of text tasks, including translation, question answering, text summarization, etc. While LLMs are unimodal, their abilities can be leveraged for multimodal understanding tasks. We propose a VQA+LLM pipeline that integrates multimodal models with LLMs for the visual question answering task. It helps the VQA model generate more accurate and fluent answers.
We open-source VLE-related resources for promoting academic research and better facilitating our community.
Try our VLE-based VQA Demo at 🤗Space 👇👇👇
Chinese LERT | Chinese and English PERT | Chinese MacBERT | ChineseMiniRBT | Chinese ELECTRA | Chinese XLNet | Chinese BERT | Knowledge distillation tool TextBrewer | Model pruning tool TextPruner
More resources released by HFL: https://github.com/iflytek/HFL-Anthology
Section | Description |
---|---|
Introduction | Introduction to VLE |
Downloads | Download links for VLE |
Comparison | Comparison of VLE with other models |
VQA with LLM | Visual question answering with LLM |
Usage | How to load VLE for different tasks |
The structure of VLE is similar to METER, which consists of two unimodal encoders for text and image separately, followed by a crossmodal fusion module. However, there are several structural differences between VLE and METER:
VLE is pre-trained with image-caption pairs. There are four objectives applied during the pre-training stage:
VLE models are pre-trained on 14M public English image-caption pairs for 25k steps with a batch size of 2048.
The following figure illustrates the VLE structure and the pre-training objectives (for simplicity, we omit the PBC objective in the figure).
The model weights are in PyTorch format and can be downloaded through the 🤗 transformers model hub. You can either download the weights and configurations manually or initialize a VLE model with from_pretrained(model_name)
method in your code. See Usage for details.
Model | Text Encoder | Image Encoder | # Params* | MODEL_NAME | Link |
---|---|---|---|---|---|
VLE-base | DeBERTa-v3-base | CLIP-ViT-base-patch16 | 378M | hfl/vle-base | link |
VLE-large | DeBERTa-v3-large | CLIP-ViT-large-patch14 | 930M | hfl/vle-large | link |
* : We exclude task heads when counting the number of parameters.
Model | Text Encoder | Image Encoder | MODEL_NAME | Link |
---|---|---|---|---|
VLE-base-for-VQA | DeBERTa-v3-base | CLIP-ViT-base-patch16 | hfl/vle-base-for-vqa | link |
VLE-large-for-VQA | DeBERTa-v3-large | CLIP-ViT-large-patch14 | hfl/vle-large-for-vqa | link |
VLE-base-for-VCR-q2a | DeBERTa-v3-base | CLIP-ViT-base-patch16 | hfl/vle-base-for-vcr-q2a | link |
VLE-large-for-VCR-q2a | DeBERTa-v3-large | CLIP-ViT-large-patch14 | hfl/vle-large-for-vcr-q2a | link |
VLE-base-for-VCR-qa2r | DeBERTa-v3-base | CLIP-ViT-base-patch16 | hfl/vle-base-for-vcr-qa2r | link |
VLE-large-for-VCR-qa2r | DeBERTa-v3-large | CLIP-ViT-large-patch14 | hfl/vle-large-for-vcr-qa2r | link |
In the following table, we compare the performance of VLE with METER and other multimodal models. The VQA results are on the test-dev set, and the VCR results are on the dev set.
Model | VQA | VCR (QA2R) | VCR (Q2A) | #Params | #PT data* |
---|---|---|---|---|---|
CoCa | 82.3 | - | - | 2.1 B | unknown |
BeiT-3 | 84.2 | - | - | 1.9 B | 21M(I-T) + 14M(I) + 160G(T) |
OFA | 82.0 | - | - | 930M | 20M(I-T) + 39M(I) + 140G(T) |
BLIP | 78.3 | - | - | 385M | ~130M(I-T) |
METER-base | 77.7 (76.8†‡) | 79.8§ | 77.6§ | 345M | 9M(I-T) |
METER-Huge | 80.3 | - | - | 878M | 20M(I-T) |
VLE-base | 77.6‡ | 83.7§ | 79.9§ | 378M | 15M(I-T) |
VLE-large | 79.3‡ | 87.5§ | 84.3§ | 930M | 15M(I-T) |
† : Result from our reimplementation.
‡ : Fine-tuning hyperparameters: lr=7e-6, batch_size={256, 512}, num_epochs=10
§ : Fine-tuning hyperparameters: lr=1e-5, batch_size=128, num_epochs=5
* : Pre-training data. I-T: Image-caption pairs. I: Images. T: Text.
From the above results, we can see that:
VLE is pre-training efficient. Compared to models with similar model sizes, VLE achieves comparable or even better performance on VQA with much less pre-training data.
VLE shows higher reasoning ability. Especially it significantly outperforms METER on Visual Commonsense Reasoning (VCR), which requires higher level language and reasoning skills than VQA.
LLMs have achieved great success on a wide range of text tasks, while the abilities of LLMs can also be leveraged for multimodal understanding tasks. Specifically, we present a VQA+LLM pipeline that integrates multimodal models with LLMs for the visual question answering task, which helps the VQA model to generate more accurate and fluent answers.
The workflows are shown in the figure below.
(a) VQA: This is the standard way to perform the VQA task with a discriminative model. The question and the image are fed into the multimodal model, and the model is trained to predict the correct answer labels.
(b) VQA + LLM: The captioning model generates a caption of the image. The caption, question, and answer candidates predicted by the VQA model are concatenated and fed to the LLM. The LLM is asked to give the most reasonable answer.
We find that VQA+LLM can not only make more accurate predictions, but also generate more fluent and readable predictions. We list some examples:
The demo is available at : https://huggingface.co/spaces/hfl/VQA_VLE_LLM
Requirements
The model classes and utilities are defined in the *.py
files in models/VLE. To import VLE into your code, just copy models directory into your project.
To run the following demo code, git clone
the repository and cd
into it, ensuring you are in the repository's root directory.
from models.VLE import VLEModel, VLEProcessor
from PIL import Image
import torch
model_name="hfl/vle-large"
images = [Image.open('pics/dogs.png')]
text = ["There are dogs on the grass."]
model = VLEModel.from_pretrained(model_name)
vle_processor = VLEProcessor.from_pretrained(model_name)
multimodal_inputs = vle_processor(text=text,images=images, return_tensors='pt',padding=True)
#forward
vle_output = model(**multimodal_inputs)
from models.VLE import VLEForVQA, VLEProcessor, VLEForVQAPipeline
from PIL import Image
model_name="hfl/vle-base-for-vqa"
text= "What is the color of the floor?"
image = Image.open("pics/door.png")
model = VLEForVQA.from_pretrained(model_name)
vle_processor = VLEProcessor.from_pretrained(model_name)
vqa_pipeline = VLEForVQAPipeline(model=model, device='cpu', vle_processor=vle_processor)
vqa_answers = vqa_pipeline(image=image, question=text, top_k=5)
print(f"Question: {text}. Answers: {vqa_answers}")
from models.VLE import VLEForITM, VLEProcessor, VLEForITMPipeline
from PIL import Image
model_dir = 'hfl/vle-base'
itm_text = ["a photo of a cat.", "a photo of dogs."]
itm_images = Image.open("pics/dogs.png")
print("Init ITM model")
model = VLEForITM.from_pretrained(model_dir)
vle_processor = VLEProcessor.from_pretrained(model_dir)
print("init ITM pipeline")
itm_pipeline = VLEForITMPipeline(model=model, device='cpu', vle_processor=vle_processor)
itm_pred = itm_pipeline([{"image": itm_images, "text": itm_text[0]},
{"image": itm_images, "text": itm_text[1]}])
for t, pred in zip(itm_text,itm_pred):
print(t,pred)
from models.VLE import VLEForPBC, VLEProcessor, VLEForPBCPipeline
from PIL import Image
model_dir = 'hfl/vle-base'
pbc_text = "pink tongues"
pbc_image = Image.open("pics/dogs.png")
print("Init PBC model")
model = VLEForPBC.from_pretrained(model_dir)
vle_processor = VLEProcessor.from_pretrained(model_dir)
print("init PBC pipeline")
pbc_pipeline = VLEForPBCPipeline(model=model, device='cpu', vle_processor=vle_processor)
pbc_pred = pbc_pipeline(image=pbc_image,text=pbc_text)
print(pbc_text)
pbc_pred['image'].save('pics/pink_tongues.png')
Please follow the instructions in examples/VCR/README.md
Please follow the instructions in examples/VQA/README.md
Welcome to follow the official WeChat account of HFL to keep up with the latest technical developments.
This repository's resources are solely intended for academic purposes, and we assume no responsibility for any unforeseen damages or losses that may result from their use.
This is not an official product by iFLYTEK Co., Ltd.
If you have questions, please submit them in a GitHub Issue.