Assigning Initial Cell States for t=0

[jxchen01]

I have a question about the cell state (or memory activation), i.e. variable c in Equation (1) of the main paper. When t=1, c_1 depends on c_0. I would like to manually assign the value of c_0 during both training and evaluation stage. Is there a way to do this?

[viorik]

Hi @jxchen01 If you look in ConvLSTM.lua, at line 134 we assign the value we want for c_0. If no value is assigned by the user (i.e. self.userPrevCell is nil) then by default c_0 is a tensor filled with 0. To specify a different tensor, you need to access this field after creating the model, with something like: model.modules[x]. .... .module.userPrevCell = myTensor I haven't tried this before, please let me know if it works like this. cheers.

[jxchen01]

Hi @viorik , I think this is exactly what I am looking for. I will try it out and post the result, if it works.

[jxchen01]

Hi @viorik , I encountered the following issue when running the forward pass. According to the error message, my understanding is that the dimension mismatch occurs when computing Ct, when step=1.

I print out the dimension of input, prevOutput, prevCell before updateOutput (before line 152). They are all of the same size, i.e. "LongTensor of 64x68x68". I am puzzled that why I still got the dimension mismatch. Also, no matter whether I assign the initial the cell state manually or all zeros, the same error always occurs.

/torch/install/share/lua/5.1/nn/CAddTable.lua:16: bad argument #2 to 'add' (sizes do not match at /tmp/luarocks_cutorch-scm-1-3474/cutorch/lib/THC/THCTensorMathPointwise.cu:113) stack traceback: [C]: in function 'add' /torch/install/share/lua/5.1/nn/CAddTable.lua:16: in function 'updateOutput' /torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'updateOutput' /torch/install/share/lua/5.1/nn/ConcatTable.lua:11: in function 'updateOutput' /torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'updateOutput' /torch/install/share/lua/5.1/nn/ConcatTable.lua:11: in function 'updateOutput' /torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'updateOutput' /torch/install/share/lua/5.1/nn/ConcatTable.lua:11: in function 'updateOutput' /torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'updateOutput' ./ConvLSTM.lua:152: in function 'updateOutput' /torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'updateOutput' /torch/install/share/lua/5.1/rnn/Recursor.lua:25: in function 'updateOutput' /torch/install/share/lua/5.1/rnn/Sequencer.lua:48: in function 'forward'

Many thanks!

[viorik]

Do you get the error even when running the original code, without any modification? If the error appears for your version, I need to see how you assign c0, what is the size of your input, and how you initialise ConvLSTM block.

[jxchen01]

@viorik , I am only using the convLSTM block of the original code. What I did was to put ConvLSTM.lua in the working directory (after installing required dependencies), and calling the ConvLSTM block by require 'ConvLSTM'. (I did not make any change in ConvLSTM.lua)

My model is very simple:

local temporal_model = nn.Sequential() local seq = nn.ConvLSTM(64, 64, 3, opt.kernalSize, opt.kernalSizeMemory, 1) seq:remember('both') seq:training() temporal_model:add(seq) temporal_model:add(nn.SpatialConvolution(64, 2, 1, 1, 1, 1, 0, 0)) temporal_model:add(nn.Transpose({1,2},{2,3})) temporal_model:add(nn.Reshape(68*68, 2)) temporal_model = nn.Sequencer(temporal_model)

which is:

nn.Sequencer @ nn.Recursor @ nn.Sequential { input -> (1) -> (2) -> (3) -> (4) -> output: nn.ConvLSTM(64 -> 64) (2): nn.SpatialConvolution(64 -> 2, 1x1) (3): nn.Transpose (4): nn.Reshape(4624x2) }

In each training iteration, the input data and target data are: Inputs: { 1 : CudaTensor - size: 64x68x68 2 : CudaTensor - size: 64x68x68 3 : CudaTensor - size: 64x68x68 } Targets: { 1 : CudaTensor - size: 4624x2 2 : CudaTensor - size: 4624x2 3 : CudaTensor - size: 4624x2 }

The code for training is as simple as:

params, gradParams = temporal_model:getParameters() params:uniform(-0.08, 0.08) temporal_model.module.module.modules[1]:initBias(1,0) for i=1, opt.nIteration do ------- prepare "inputs" and "targets" here ------- \ suppose inputs and targets are tables of aforementioned size **

    local feval = function (x)
          if x ~= params then params:copy(x) end
          gradParams:zero()

            ------ forward -------
    local outputs=temporal_model:forward(inputs)
    local err = criterion:forward(outputs,targets)

    ------ backward ------
    local gradOutputs = criterion:backward(outputs,targets)
    local gradInputs = temporal_model:backward(inputs, gradOutputs)

    if opt.clip>0 then
        gradParams:clamp(-opt.clip, opt.clip)
    end

    return err, gradParams
end

local _, loss = optim.adam(feval, params, optim_config)

end

[viorik]

I am not able to reproduce the problem. I have cloned a fresh copy of my repo, and used this test file:

require 'cunn'
require 'rnn'
require 'ConvLSTM'

local temporal_model = nn.Sequential()
local seq = nn.ConvLSTM(64, 64, 3, 7, 7, 1)
seq:remember('both')
seq:training()
temporal_model:add(seq)
temporal_model:add(nn.SpatialConvolution(64, 2, 1, 1,1,1,0,0))
temporal_model:add(nn.Transpose({1,2},{2,3}))
temporal_model:add(nn.Reshape(68*68, 2))
temporal_model = nn.Sequencer(temporal_model)
temporal_model:cuda()

print (temporal_model)

input = torch.Tensor(64,68,68):cuda()

inputTable = {}
table.insert(inputTable,input)
table.insert(inputTable,input)
table.insert(inputTable,input)

oo = temporal_model:forward(inputTable)

print (#oo)

The output is, as expected 3, the size of the output table. Can you spot any difference?

[jxchen01]

Hi @viorik , I found the reason. The error of my program was resulted from opt.kernalSizeMemory, where I used a bad value "10". It should be an odd number, right? If an even value is assigned as the kernel size, the value of self.padc or self.padm (line 20 and 21) will cause the dimension mismatch after the convolution.

Really appreciate your help!