Query regarding Pretrained model

[prajjwal1]

Hi,

In Pre-training VL-BERT section, you've highlighed some tasks on which model was trained.

1. Pretraining on Conceptual Conceptions
2. Masked Language Modeling with Visual Clues
3. Masked RoI Classification with Linguistic Clues

I had some questions:

1. Is the pretrained model which you've provided was trained on all 3 objectives ? If yes, should it be used for finetuning now ? If not, do I need to perform pretrainining ?
2. Can you please clarify the order of usage of datasets in all three objectives ? (As in in which dataset was used to satisfy the objective)
3. In the code, I'm seeing that the pretrained models dict partially updates the weights of the instantiated model. Why so ? Are there some parts of the model that are initialized as mentioned in the paper ?

[jackroos]

1. Actually there are only two tasks in pre-training: Masked Language Modeling with Visual Clues, Masked RoI Classification with Linguistic Clues. Sure, the pre-trained models provided in this repo have been pre-trained on these tasks, you don't need to perform pre-training by yourself.
2. We use Conceptual Captions for both tasks, and English Wikipedia & BookCorpus only for Masked Language Modeling task.
3. Could you provide the detailed link to the line of code since I am not sure where do you exactly mean?

Thanks!

[prajjwal1]

Thanks for the reply.

3. For (3), I meant this line of code. Is it only partially loading the weights ?

I have some more questions.

4. Could you also share the weights for fast rcnn module ? It seems that the pretrained model is meant for ResNetVLBert only.
5. Do I need to train fast rcnn on Visual Genome for my task?
6. I want to port fast rcnn to torchvision's faster rcnn. What does obj_reps actually represent here, are these the box predictions coming out of ROI Head ? And you seem to be using 'cnn_regularization' loss which is cross entropy loss for classes, what about MSE loss for regression of bounding boxes ? I think MSE loss would be necessary for improving region proposals.
7. If I were to port to torchvision's faster_rcnn implementation, please see here, Will detections from self.transform.postprocess be 'obj_reps' and cross entropy loss be 'cnn_regularization' loss ?

[jackroos]

@prajjwal1

3. The partial loading is because downstream task and pre-training task may have different prediction heads, and only common weights (including the Fast RCNN, VL-BERT, and maybe some common heads) are loaded.
4. Actually, the weights of Fast RCNN is included: https://github.com/jackroos/VL-BERT/blob/28cbde93de9e9670fdcdf2b75af2855dafe1e280/pretrain/modules/resnet_vlbert_for_pretraining_multitask.py#L19 5/6/7. Here I think you have some misunderstanding about Fast RCNN in our model, it's not Faster RCNN, so there is no RPN in it. Actually, our workflow is:
   • (Offline) Use the pre-trained Faster RCNN to extract bounding boxes and store them (refer to https://github.com/jackroos/VL-BERT/blob/28cbde93de9e9670fdcdf2b75af2855dafe1e280/data/conceptual-captions/ReadMe.txt#L59);
   • Use only the backbone (ResNet) and RoI head of pre-trained Faster RCNN to initialize our Fast RCNN;
   • (Online) Treat precomputed boxes as region proposals, and using Fast RCNN to extract their features.

P.S. The obj_reps refers to visual feature of each RoI used in VL-BERT, obj_reps_raw means RoI features coming out of RoI head, and the cnn_regularization is deprecated, we don't use it in both pre-training and fine-tuning.