search

allenai / allennlp

An open-source NLP research library, built on PyTorch.
http://www.allennlp.org
Apache License 2.0
11.73k stars 2.24k forks source link

SeqDecoder abstraction and Variational Autoencoders #5021

Closed vikigenius closed 3 years ago

vikigenius commented 3 years ago

I was taking a look at the composed seq2seq models and their abstractions. Is the SeqDecoder suitable for implementing Variational Autoencoders for generation tasks ?

Because it seems like SeqDecoder assumes that the encoder output in forward is the output from a Seq2SeqEncoder, however in a Variational Autoencoder wouldn't the better option for an encoder be a Seq2VecEncoder ?

The documentation does not make it clear what the encoder_out dictionary is supposed to be.

epwalsh commented 3 years ago

Hey @vikigenius, I think encoder_out can be anything as long as your DecoderNet can handle it.

epwalsh commented 3 years ago

I'm not super familiar with this API though and like you said, it was probably not intended to be used with VAEs. But so far I don't see any reason why it can't be.

Either way, the documentation could definitely be improved.

github-actions[bot] commented 3 years ago

This issue is being closed due to lack of activity. If you think it still needs to be addressed, please comment on this thread 👇

vikigenius commented 3 years ago

I just revisited this and found out why the current abstraction cannot be used with VAEs> It is because, a VAE encoder is expected to produce the following shape. [batch_size, latent_dim] equivalent to a Seq2Vec encoder. However the SeqDecoder abstraction does the following

final_encoder_output = util.get_final_encoder_states(
    encoder_out["encoder_outputs"],
    encoder_out["source_mask"],
    bidirectional=self._bidirectional_input,
)

Which expects encoder_outputs to be from a Seq2Seq encoder with shape [batch_size, num_tokens encoder_output_dim] which is 3d and thus fails.

A custom decoder_net implementation to handle it probably should work.