search

seongminp / transformers-into-vaes

Code for "Finetuning Pretrained Transformers into Variational Autoencoders"
https://aclanthology.org/2021.insights-1.5
36 stars 11 forks source link

passing projected latent space #1

Open safakkbilici opened 3 years ago

safakkbilici commented 3 years ago

Thanks for this nice work and reproducible code. If I understood your approach well, I think I am missing the code line that you passed latent space z to decoder as encoder_hidden_states. I will be glad if you point that line. Thanks.

seongminp commented 3 years ago

Hello. Latent z is passed as past_key_values instead of encoder_hidden_states of the decoder:

https://github.com/seongminp/transformers-into-vaes/blob/16205c8da8731b0097d80eeca219a878e0397beb/vendor_t5.py#L134

I think encoder_hidden_states is a more natural choice, but using past_key_values required fewer modifications of transformers library. More importantly the OPTIMUS paper uses past_key_values so that's why I went with that.

z is rearranged before being fed into the decoder here: https://github.com/seongminp/transformers-into-vaes/blob/16205c8da8731b0097d80eeca219a878e0397beb/vendor_t5.py#L226

safakkbilici commented 3 years ago

Oh I got it. Thank you for your kind response.