nicholas-leonard
commented
8 years ago Hi @hsheil. I like where this is going. I am reproducing your code here :
require 'rnn'
function build_network(inputSize, hiddenSize, outputSize)
-- I1: add in a dropout layer
rnn = nn.Sequential()
:add(nn.Sequencer(nn.Linear(inputSize, hiddenSize)))
:add(nn.Sequencer(nn.LSTM(hiddenSize, hiddenSize)))
:add(nn.Sequencer(nn.LSTM(hiddenSize, hiddenSize)))
:add(nn.Sequencer(nn.Linear(hiddenSize, outputSize)))
:add(nn.Sequencer(nn.LogSoftMax()))
-- I1: Adding this line makes the loss oscillate a lot more during training, when according to
-- http://arxiv.org/abs/1409.2329 this should *help* model performance
-- A1: initialization often depends on each dataset.
--rnn:getParameters():uniform(-0.1, 0.1)
return rnn
end
-- Keep the input layer small so the model trains / converges quickly while training
local inputSize = 10
-- Most models seem to use 512 LSTM units in the hidden layers, so let's stick with this
local hiddenSize = 512
-- We want the network to classify the inputs using a one-hot representation of the outputs
local outputSize = 3
local rnn = build_network(inputSize, hiddenSize, outputSize)
--artificially small batchSize again for easy training
-- this can be the number of sequences to train on
local batchSize=5
-- the dataset size is the length of each of the batchSize sequences.
local dsSize=20
-- number of classes
local nClass = 10
inputs = {}
targets = {}
-- Build up our inputs and targets
-- I2, add code so that if --cuda supplied, these become CudaTensors
-- using the opt.XXX and 'require cunn'
-- I3 - replace this random data set with something more meaningful / learnable
-- and with a realistic testing and validation set
for i = 1, dsSize do
table.insert(inputs, torch.randn(batchSize,inputSize))
table.insert(targets, torch.LongTensor(batchSize):random(1,nClass))
end
-- Decorate the regular nn Criterion with a SequencerCriterion as this simplifies training quite a bit
seqC = nn.SequencerCriterion(nn.ClassNLLCriterion())
local count = 0
local numEpochs=100
local start = torch.tic()
--Now let's train our network on the small, fake dataset we generated earlier
while numEpochs ~= 0 do
rnn:training()
count = count + 1
out = rnn:forward(inputs) -- your are feeding batchSize sequences each of length dsSize steps
err = seqC:forward(out, targets)
gradOut = seqC:backward(out, targets)
rnn:backward(inputs, gradOut)
local currT = torch.toc(start)
print('loss', err .. ' in ', currT .. ' s')
--TODO, make this configurable / reduce over time as the model converges
rnn:updateParameters(0.05)
-- I5: Are these steps necessary? Seem to make no difference to convergence if called or not
-- Perhaps they are being called by
rnn:zeroGradParameters()
-- rnn:forget() dont need this as Sequencer handle it directly.
start = torch.tic()
-- I6: Make this configurable based on the convergence, so we keep going for bigger, more complex models until they are trained
-- to an acceptable accuracy
-- Also add in code to save out the model file to disk for evaluation / usage externally periodically
numEpochs = numEpochs - 1
endSo I modified a couple of things. In its current (above) form, each epoch the rnn sees the entirety of the dataset. For a real dataset, you would need to add another inner loop where your split the batchSize x dsSize data into chunks of smaller batches of sequences : batchSize x seqLength where seqLength << dsSize.
jundengdeng
jrich9999
hsheil
rracinskij
hughperkins
kmul00
beldaz
Hi
I'm trying to document the simplest LSTM example possible. By "simplest", I mean the intersection of least lines of code combined with minimal use of libraries that hide away details. Once this simple case is working well, then the plan is to add in libraries like dp to show what that library provides, i.e. dp.Experiment. And to do this in a progressive way so that the reader can roll forwards / backwards between versions of the code to build understanding. So:
The code in the gist below works (loss reduces as the model trains), but with some caveats which I've documented as inline comments, numbered I0 through I6. I4 is the biggest issue i'd like to resolve..
I'd appreciate any feedback on the specific items in the code (esp. I4) and also any other comments on the code! When this code is good enough, the plan is to publish it as a tutorial of sorts on the rnn package with worked examples that progressively become more complex / advanced - feeding into other issue requests in this repo for more examples. Thanks in advance!
Here's the gist: https://gist.github.com/hsheil/54c0e83d4666db5081df