use your model with your weight

[masoudpz]

How can i use your code for loading model and weights for my input videos?

[ClementPinard]

Here is the simplest snippet I can come up with to use the caffe pretrained model :

torch image version

require 'image'
require 'nn'
require 'nngraph'
require 'cutorch'
require 'cunn'
require 'cudnn'

local model = torch.load('path_to_pretrained')

--not necessary but recommendend
model = cudnn.convert(model:cuda(),cudnn)

-- Pictures are already in the range [0,1]
-- so you don't need to normalize it for caffe pretrained)
local img1 = image.load('path_to_img1'))
local img2 = image.load('path_to_img2')) 

local input = torch.CudaTensor(1,6,img1:size(2),img1:size(3))
input[{{},{1,3}}]:copy(img1)
input[{{},{4,6}}]:copy(img2)

model:forward(input)

opencv torch version

local cv = require 'cv'
require 'cv.imgcodecs'

require 'nn'
require 'nngraph'
require 'cutorch'
require 'cunn'
require 'cudnn'

local model = torch.load('path_to_pretrained')

--not necessary but recommendend
model = cudnn.convert(model:cuda(),cudnn)

--permute HxWxC to CxHxW (3 planes of floats instead of a plane of 3D vectors)
local img1 = cv.imread('path_to_img1'):permute(3,2,1):float()/255 
local img2 = cv.imread('path_to_img2'):permute(3,2,1):float()/255

local input = torch.CudaTensor(1,6,img1:size(2),img1:size(3))
input[{{},{1,3}}]:copy(img1)
input[{{},{4,6}}]:copy(img2)

model:forward(input)

Alos it is important to have images sized with multiples of 64, e.g 128x256 or 512x768 You can use image.resize and image.crop to do that.

[masoudpz]

I used your code, there is one error

/bin/luajit: flownetModel.lua:19: '}' expected near ']' stack traceback: [C]: in function 'dofile' ...asin/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk [C]: at 0x00406670

and i changed 2 lines into

input[{{},{1,3}}]:copy(img1) input[{{},{4,6}}]:copy(img2)

and after this changes

install/bin/luajit: torch/install/share/lua/5.1/nn/THNN.lua:110: bad argument #3 to 'v' (cannot convert 'struct THFloatTensor ' to 'struct THCudaTensor ') stack traceback: [C]: in function 'v' /torch/install/share/lua/5.1/nn/THNN.lua:110: in function 'SpatialConvolutionMM_updateOutput' ...i/torch/install/share/lua/5.1/nn/SpatialConvolution.lua:79: in function 'func' torch/install/share/lua/5.1/nngraph/gmodule.lua:345: in function 'neteval' torch/install/share/lua/5.1/nngraph/gmodule.lua:380: in function 'forward' flownetModel.lua:22: in main chunk [C]: in function 'dofile' /torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk [C]: at 0x00406670 @ClementPinard thanks for your response

I loaded flowNetS.t7 as my model from your google drive link.

[ClementPinard]

Sorry About that typo, i updated my first comment. The error is saying that you are either feeding the network FloatTensors (while it expects cuda tensors) or the network has not been converted. Most likely the network conversion failed. Make sure you have cutorch and cudnn installed right.

You can try to replace model = cudnn.convert(model:cuda(),cudnn) with just model = model:cuda() which should work but will be slower (about half speed)

[masoudpz]

Thanks alot.it works! for last question How can i save output of network as an image ?

[ClementPinard]

From liveDemo.lua

    local output = net:forward(input):float() --if using torch pretrained use net:forward(input)[5]:float() to select last scale
    local output_ = torch.FloatTensor(3,output:size(2),output:size(3))
    output_[1]:fill(1)
    local saturation = 100 --flow is saturated at 100
    output_[{{2,3}}]:copy(output:clamp(-saturation,saturation)/saturation)
    local to_save = image.yuv2rgb(output_)
    image.save('path',to_save)

[masoudpz]

require 'image'
require 'nn'
require 'nngraph'
require 'cutorch'
require 'cunn'
require 'cudnn'

local model = torch.load('flowNetS.t7')

--not necessary but recommendend
model = model:cuda()

-- Pictures are already in the range [0,1]
-- so you don't need to normalize it for caffe pretrained)
local img1 = image.load('1.jpg')
local img2 = image.load('2.jpg') 

local input = torch.CudaTensor(1,6,img1:size(2),img1:size(3))
input[{{},{1,3}}]:copy(img1)
input[{{},{4,6}}]:copy(img2)

local output = model:forward(input)[5]:float() 
local output_ = torch.FloatTensor(3,output:size(2),output:size(3))
output_[1]:fill(1)
local saturation = 100 --flow is saturated at 100
output_[{{2,3}}]:copy(output:clamp(-saturation,saturation)/saturation)
local to_save = image.yuv2rgb(output_)
image.save('res.jpg',to_save)

this is my final code and now i see

/torch/install/bin/luajit: bad argument #2 to '?' (out of range) stack traceback: [C]: at 0x7f5334f7ac70 [C]: in function '__index' flownetModel.lua:24: in main chunk [C]: in function 'dofile' ...asin/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk [C]: at 0x00406670

@ClementPinard

[ClementPinard]

it's because pretrained flowNet model only delivers highest scale in a sole tensor, while models from scratch deliver the 5 scales in table of tensors. To avoid problems like this use this :

local output = model:forward(input)
if torch.type(output) == 'table' then
  output = output[5]:float()
else
  output = output:float()

[masoudpz]

Hi again! :) and thanks for your response.

require 'image'
require 'nn'
require 'nngraph'
require 'cutorch'
require 'cunn'
require 'cudnn'

local model = torch.load('flowNetS.t7')

--not necessary but recommendend
model = model:cuda()

-- Pictures are already in the range [0,1]
-- so you don't need to normalize it for caffe pretrained)
local img1 = image.load('1.jpg')
local img2 = image.load('2.jpg')

local input = torch.CudaTensor(1,6,img1:size(2),img1:size(3))
input[{{},{1,3}}]:copy(img1)
input[{{},{4,6}}]:copy(img2)

local output = model:forward(input)
if torch.type(output) == 'table' then
  output = output[5]:float()

else
  output = output:float()
end

local output_ = torch.FloatTensor(3,output:size(2),output:size(3))
output_[1]:fill(1)
local saturation = 100 --flow is saturated at 100
output_[{{2,3}}]:copy(output:clamp(-saturation,saturation)/saturation)
local to_save = image.yuv2rgb(output_)
image.save('3.jpg',to_save)

this is my final code, and now i get this error

.lua:37: inconsistent tensor size at /torch/lib/TH/generic/THTensorCopy.c:7 stack traceback: [C]: in function 'copy' masoud2.lua:37: in main chunk [C]: in function 'dofile' ...asin/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk [C]: at 0x00406670 in this line

output_[{{2,3}}]:copy(output:clamp(-saturation,saturation)/saturation);

[ClementPinard]

what line 37 is referring to ? I advise you to call just before line 37

print({tensor1,tensor2})

with tensor1and tensor2 the two tensors involved in the operation that triggers an error to see what's going on