All modules in the paper have been done. Code can run without error. Only support VRD dataset now.
There is still something wrong with training loss for original code in FactorizableNe. If I train the model with only RPN pretrained, all proposals will be predicted as 0(background) after one iteration, with or without added modules in this repo.