senarvi / theanolm

TheanoLM is a recurrent neural network language modeling tool implemented using Theano
Apache License 2.0
81 stars 29 forks source link

Create a binary representation of the training and validation data #3

Closed senarvi closed 6 years ago

senarvi commented 8 years ago

Gzipped training and validation data cannot be used directly. They will not necessarily fit into memory, so they should be extracted to a different file. To improve training time, they should be converted to word IDs. Also transfer input to GPU in bigger chunks, then use givens to point to the correct mini-batch.