ALBEF: Align Before Fuse

[ggoggam]

Model description

Align Before Fuse (ALBEF) is a vision-language (VL) model that showed competitive results in numerous VL tasks such as image-text retrieval, visual question answering, visual entailment, and visual grounding.

The authors propose to use text encoder (BERT's first half layers) and image encoder (ViT) to create an aligned representation for respective modality before fusing them together with a multi-modal encoder (BERT's second half layers). The model is trained on multi-modal representation tasks and momentum distillation to achieve state-of-the-art results in VL tasks.

As multi-modal models are gaining more attention in academia/industry, I think ALBEF could be a nice addition to the transformers library.

Open source status

• [X] The model implementation is available
• [X] The model weights are available

Provide useful links for the implementation

• There are an official implementation and pre-trained/fine-tuned weights by the authors at this repo
• Link to the paper

[DanielFLevine-zz]

@jkgrad What is the state of this issue? If no one is working on this, I would like to implement it.

[LysandreJik]

Hey @DanielFLevine, we'd love for you to try and contribute that model!

cc @NielsRogge who can help out once he's back from leave :)

[DanielFLevine-zz]

@LysandreJik @NielsRogge Great! I've already started looking over the authors' code. Will reach out with any questions.

[ethansmith2000]

Is there still interest for this?

[Yassin-fan]

Same question

[amyeroberts]

@DanielFLevine-zz - any updates on the model port?

[chengjiali]

any progress on this?