It would be great to have an explicit keras.embeddings.Embedding layer for the text input instead of a TimeDistributedDense layer. An Embedding layer for the text input would let us initialise the model with word2vec or GloVe vectors. This should make it easier to train models because we wouldn't need to also learn the word representations. Another side-effect of using the TimeDistributedDense layer is the size of the word representations is tied to the size of the hidden layer.
Fixing this issue would require a rewrite of how the data_generator builds the text input and text targets, and then changing the model so it has an Embedding layer instead of a TimeDistributedDense layer.
@scfrank if you agree this makes sense, could you take a look into this?
It would be great to have an explicit
keras.embeddings.Embeddinglayer for the text input instead of aTimeDistributedDenselayer. AnEmbeddinglayer for the text input would let us initialise the model with word2vec or GloVe vectors. This should make it easier to train models because we wouldn't need to also learn the word representations. Another side-effect of using theTimeDistributedDenselayer is the size of the word representations is tied to the size of the hidden layer.Fixing this issue would require a rewrite of how the
data_generatorbuilds the text input and text targets, and then changing the model so it has anEmbeddinglayer instead of aTimeDistributedDenselayer.@scfrank if you agree this makes sense, could you take a look into this?