Convolutional LSTM

[ankurhanda]

I'm implementing convolution LSTMs with autograd - the pseudo code is here https://github.com/ankurhanda/convlstm.autograd/blob/master/RecurrentConvLSTMNetwork.lua. The code is adapted from RecurrentLSTMNetwork.lua in src/model which is very nicely structured. However, I always get bad argument #1 (field weight does not exist) in function 'conv_ixinput' when I test my RecurrentConvLSTMNetwork.lua. I wonder if nn.SpatialConvolution allows conv(x, W, b). Otherwise what's the best way to implement this while keeping the structure similar to RecurrentLSTMNetwork?

[alexbw]

@ankurhanda Interesting. Could you provide a distilled and runnable example that we can include in our test suite? Then we'll do our best to get to the bottom of this.

[ankurhanda]

here is the test code I wrote

 Models_RecurrentConvLSTMNetwork = function()

        local f, params = autograd.model.RecurrentConvLSTMNetwork({
                    inputFeatures = 2,
                    hiddenFeatures = 4,
                    outputType = 'last',
                    })

        local loss = function(params, input,target)
                local v = f(params,input)
                local mse_error =  autograd.nn.MSECriterion()
                return mse_error(v,target)
        end

      -- Test on sequence data:
     local i = torch.randn(2, 4, 4)
     local target = torch.randn(4,2,2)
     local o = loss(params, i,target)
     local g = autograd(loss)(params, i,target)

      -- Checks
     tester:asserteq(type(g), 'table', 'gradients could not be computed')

      -- Gradcheck:
     tester:assert(gradcheck(loss, params, i,target), 'incorrect gradients')
  end,

I added this test within test/test.lua. However, it crashes well before in the convLSTM function where it is trying to convolve --- hence that bad argument error. Upon a quick google search, I initially thought this error had something to do with old version of nn so I updated the nn as well as torch but it still crashes.

Moreover, I see common.lua in src/model where a sequencer is doing pretty much the same job of calling the function like conv(x,W,b) and that seems to pass the test so I'm slightly puzzled why it crashes. I may have misunderstood something here.

There's my other code which does convolution/deconvolution without bias and has been tested https://github.com/ankurhanda/culstmextra. This is needed for LSTMs and I can add once I have this running properly. I foresee that people will use convolutionLSTMs in the coming months like we have been doing now, to understand videos so it will be nice to have an implementation with autodifferentiation.

[alexbw]

Try printing params.w_ixinput, or any of your named params. They show up as nil, because you have not assigned them names in your params table. You pass them in as a list here, which means the keys will be numbers, not strings: https://github.com/ankurhanda/convlstm.autograd/blob/master/RecurrentConvLSTMNetwork.lua#L56-L73. So the function call fails because you pass in nil to these functions.

Let me know if you find any issues specific to autograd, we'll be happy to help.