search

airsplay / lxmert

PyTorch code for EMNLP 2019 paper "LXMERT: Learning Cross-Modality Encoder Representations from Transformers".
MIT License
923 stars 157 forks source link

Get embeddings of language encoder and Object-Relationship Encoder? #94

Closed yezhengli-Mr9 closed 3 years ago

yezhengli-Mr9 commented 3 years ago

Any suggestion of getting embeddings of language encoder and Object-Relationship Encoder? I refers to output_hidden_state mentioned in this blog as well as the official tutorial.

zhmd commented 3 years ago

I'm not 100% sure, but if you refer to the encoded representation before the cross-modality interaction module, then probably you need to hack these two lines and save these intermediate representations: https://github.com/airsplay/lxmert/blob/0db1182b9030da3ce41f17717cc628e1cd0a95d5/src/lxrt/modeling.py#L554-L559

If you need the ones after the cross-modality interaction module, then just take these two: https://github.com/airsplay/lxmert/blob/0db1182b9030da3ce41f17717cc628e1cd0a95d5/src/lxrt/modeling.py#L566

yezhengli-Mr9 commented 3 years ago

I'm not 100% sure, but if you refer to the encoded representation before the cross-modality interaction module, then probably you need to hack these two lines and save these intermediate representations:

https://github.com/airsplay/lxmert/blob/0db1182b9030da3ce41f17717cc628e1cd0a95d5/src/lxrt/modeling.py#L554-L559

If you need the ones after the cross-modality interaction module, then just take these two:

https://github.com/airsplay/lxmert/blob/0db1182b9030da3ce41f17717cc628e1cd0a95d5/src/lxrt/modeling.py#L566

Sort of help, thanks! @zhmd