Add Context AutoEncoder (CAE)

[charlesCXK]

Model description

The corresponding paper has been accepted by International Journal of Computer Vision (IJCV).

We present a novel masked image modeling (MIM) approach, context autoencoder (CAE), for self-supervised representation pretraining. We pretrain an encoder by making predictions in the encoded representation space. The pretraining tasks include two tasks: masked representation prediction - predict the representations for the masked patches, and masked patch reconstruction - reconstruct the masked patches. The network is an encoder-regressor-decoder architecture: the encoder takes the visible patches as input; the regressor predicts the representations of the masked patches, which are expected to be aligned with the representations computed from the encoder, using the representations of visible patches and the positions of visible and masked patches; the decoder reconstructs the masked patches from the predicted encoded representations. The CAE design encourages the separation of learning the encoder (representation) from completing the pertaining tasks: masked representation prediction and masked patch reconstruction tasks, and making predictions in the encoded representation space empirically shows the benefit to representation learning. We demonstrate the effectiveness of our CAE through superior transfer performance in downstream tasks: semantic segmentation, object detection and instance segmentation, and classification.

Open source status

• [X] The model implementation is available
• [X] The model weights are available

Provide useful links for the implementation

Code link: https://github.com/Atten4Vis/CAE Author: @charlesCXK Paper link: https://arxiv.org/abs/2202.03026

[charlesCXK]

I would like to work towards adding this model to huggingface transformers @NielsRogge

[ArthurZucker]

cc @amyeroberts @rafaelpadilla

[amyeroberts]

Hi @charlesCXK, thanks for opening this model request!

The easiest and recommended way to make a model available in transformers is to add the modeling code directly on the hub: https://huggingface.co/docs/transformers/custom_models. This means, once working, the model can be found and used immediately without having to go through the PR process. We find this is a lot quicker as the bar for adding code into the library is high due to the maintenance cost of every new model, and so reviews take quite a while.