search

IBM / pytorch-seq2seq

An open source framework for seq2seq models in PyTorch.
https://ibm.github.io/pytorch-seq2seq/public/index.html
Apache License 2.0
1.5k stars 376 forks source link

Dropout error using external embeddings #171

Closed rrkarim closed 6 years ago

rrkarim commented 6 years ago

Training model with fasttext-en embedding with hidden size of 300 throws dropout error: UserWarning: dropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout=0.2 and num_layers=1. Maybe there is need of adjusting embedding hidden sizes.

pskrunner14 commented 6 years ago

@CoderINusE try increasing the number of layers in your model. PyTorch doesn't allow adding droupout in a single layer RNN. I'm not sure what it has to do with fasttext embeddings.

rrkarim commented 6 years ago

I'm not changing number of layers in the model. I just have my pre-trained vectors of English words with hidden size of 300. Then I load them into input_vocab and output_vocab using torchtext.vocab.FastText with max_vectors=5000 and trying to overload embedding and update_embedding. The only file that I'm changing is the examples/sample.py. If it gives me the error and I should change something in the architecture than it is the bug or it should be stated in README.

pskrunner14 commented 6 years ago

@CoderINusE it is not a bug. This is a framework written in PyTorch and PyTorch does'nt allow applying dropout if there is only one layer (last layer) in the RNN as stated in the docs under dropout option. You could also have a look at the source code where the warning is explicitly mentioned:

https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/rnn.py#L43

and also on our develop branch:

https://github.com/IBM/pytorch-seq2seq/blob/develop/seq2seq/models/base_rnn.py#L45