facebookresearch / mmf

A modular framework for vision & language multimodal research from Facebook AI Research (FAIR)
https://mmf.sh/
Other
5.48k stars 933 forks source link

[feature] Add fusion techniques #162

Closed apsdehal closed 4 years ago

apsdehal commented 5 years ago

🚀 Feature

Fusion techniques for multimodal fusion are an important component of VQA models. Remi's repo https://github.com/Cadene/block.bootstrap.pytorch has most of them implemented beforehand. We can add it as a dependency and import whatever is required.

pushkalkatara commented 4 years ago

Hi, can I work on this issue? I notice that in the models fusion layers are imported from modules/layers.py. Do these need to be replaced by remi's repo?

apsdehal commented 4 years ago

Hi, Thanks for working on this. You don't have to replace that file. Instead, you have to expose layers from Remi's repo into this file. So, that one can directly import these from pythia. It might be better to create a new file in modules folder named fusions