performance of Flownet2 is not good as the examples

[Jingyao12]

Hi, I am trying to use the Flownet2. I installed it and I run run-flownet.py /path/to/FlowNet2/FlowNet2t_weights.caffemodel.h5 \ /path/to/$net/$net_deploy.prototxt.template \ x.png y.png z.flo using the example images in flownet2/data/FlyingChairs_examples and then I got the error Layer conv1 has unknown engine after that I replace the "engine: CUDNN" by "engine: CAFFE" in prototxt files to fix this problem. Then I used FlowNet2 on the example images got the .flo file. I compare it with the results in example files. My results is much bluring and cannot see chair in the output. Could you help me figure out this problem? or I missed some necessary preprocess?

[nikolausmayer]

Hi,

the "FlowNet2" model's output and the groundtruth for the example pair (0000000-img0.ppm, 0000000-img1.ppm) are:

Sure the output is not as good as the groundtruth, but the chairs are recognizable. Do you get a much worse result?

[Jingyao12]

Hi,

Thanks you for the reply. I tried more times and now I can get more robust results. I still have another question. Actually the output is 3-D matrix and 2 channels in the 3rd dimension. The 2 images along the 3rd dimension are similar to the first input image and the intensity value are different from each other. How can I understand this two images attached?

Thank you !

Best, Jingyao

From: Nikolaus Mayer [mailto:notifications@github.com] Sent: Friday, October 27, 2017 5:45 AM To: lmb-freiburg/flownet2 Cc: Li, Jingyao; Author Subject: Re: [lmb-freiburg/flownet2] performance of Flownet2 is not good as the examples (#77)

Hi,

the "FlowNet2" model's output and the groundtruth for the example pair (0000000-img0.ppm, 0000000-img1.ppm) are:

[Image removed by sender. flow]https://urldefense.proofpoint.com/v2/url?u=https-3A__user-2Dimages.githubusercontent.com_7045923_32098139-2D09e06ad2-2Dbb0c-2D11e7-2D95db-2Dc716c49461cd.png&d=DwMFaQ&c=ZbgFmJjg4pdtrnL2HUJUDw&r=iPwEFBadX7fVtSOw208U6xhNNuaT4uFigfdW6CFf260&m=3If3vCZbU0dqjNbQ63ewnxw_el1tbYsojdi_tzWmKWA&s=vtbec_XlE75P7Zgo73635On5Ul1JdGOZ6Nfl3JwZNIY&e= [Image removed by sender. gt]https://urldefense.proofpoint.com/v2/url?u=https-3A__user-2Dimages.githubusercontent.com_7045923_32098148-2D1168dc58-2Dbb0c-2D11e7-2D90e6-2Dba6514a34389.png&d=DwMFaQ&c=ZbgFmJjg4pdtrnL2HUJUDw&r=iPwEFBadX7fVtSOw208U6xhNNuaT4uFigfdW6CFf260&m=3If3vCZbU0dqjNbQ63ewnxw_el1tbYsojdi_tzWmKWA&s=gOlpLHU6KwkxfQiH_wgjMv-btxCdxu8tj6EqqemZx6c&e=

Sure the output is not as good as the groundtruth, but the chairs are recognizable. Do you get a much worse result?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_lmb-2Dfreiburg_flownet2_issues_77-23issuecomment-2D339925751&d=DwMFaQ&c=ZbgFmJjg4pdtrnL2HUJUDw&r=iPwEFBadX7fVtSOw208U6xhNNuaT4uFigfdW6CFf260&m=3If3vCZbU0dqjNbQ63ewnxw_el1tbYsojdi_tzWmKWA&s=aPPgXdC6FTwC2ch8JB-OISORS6mkJmmIaBJMfWNkapM&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AV1iUngiwSXO31zlqdcWJtAyw693G0Tvks5swaYngaJpZM4QIEwk&d=DwMFaQ&c=ZbgFmJjg4pdtrnL2HUJUDw&r=iPwEFBadX7fVtSOw208U6xhNNuaT4uFigfdW6CFf260&m=3If3vCZbU0dqjNbQ63ewnxw_el1tbYsojdi_tzWmKWA&s=gdDlQKTH4YMPTt1fu_INo90AvdWjrmf0KS1elR6BhRo&e=.

[nikolausmayer]

I'm not sure I understand what you mean. The images I attached are just visualizations of the estimated and the groundtruth optical flow.

[Jingyao12]

Hi

My question is based on the result file .flo. For example, the put image size is 370370 then the output z.flo file size is 3703702. How can I understand this output file, two 370370 images. Please see the attached images. The legend is as follow:

Output[:,:,0] 0000000-img0.ppm

Output[:,:,1] 0000000-img1.ppm

From: Nikolaus Mayer [mailto:notifications@github.com] Sent: Friday, October 27, 2017 12:00 PM To: lmb-freiburg/flownet2 Cc: Li, Jingyao; Author Subject: Re: [lmb-freiburg/flownet2] performance of Flownet2 is not good as the examples (#77)

I'm not sure I understand what you mean. The images I attached are just visualizations of the estimated and the groundtruth optical flow.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_lmb-2Dfreiburg_flownet2_issues_77-23issuecomment-2D340012746&d=DwMCaQ&c=ZbgFmJjg4pdtrnL2HUJUDw&r=iPwEFBadX7fVtSOw208U6xhNNuaT4uFigfdW6CFf260&m=fjNmcs1gHJPs7V_fT4rtBNj8uyu7DXi7jZBauHHGU-U&s=OhENXd-3pevnXGIPODvqmOBMNKd0EY-uAHMWHypjxZ4&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AV1iUizkf69QAOzUrtP39aSRHkdeFBPGks5swf3ogaJpZM4QIEwk&d=DwMCaQ&c=ZbgFmJjg4pdtrnL2HUJUDw&r=iPwEFBadX7fVtSOw208U6xhNNuaT4uFigfdW6CFf260&m=fjNmcs1gHJPs7V_fT4rtBNj8uyu7DXi7jZBauHHGU-U&s=HoPqwBERA1T4pootAXSDf54JRmS6DEdkX4tDstVWFvQ&e=.

[nikolausmayer]

Ah, I see. The first layer of the network output contains the x-component (the "horizontal") of the optical flow, and the second layer contains the y-component ("vertical").

The flow vector for a pixel (x,y) is (Output[x,y,0], Output[x,y,1]).

[Jingyao12]

Thank you for the reply. May I have one more question? How can I plot the optical flow using *.flo and get the results which have the same color coding as you shown before?

From: Nikolaus Mayer [mailto:notifications@github.com] Sent: Saturday, October 28, 2017 5:41 AM To: lmb-freiburg/flownet2 flownet2@noreply.github.com Cc: Li, Jingyao jingyao.li@novartis.com; Author author@noreply.github.com Subject: Re: [lmb-freiburg/flownet2] performance of Flownet2 is not good as the examples (#77)

Ah, I see. The first layer of the network output contains the x-component (the "horizontal") of the optical flow, and the second layer contains the y-component ("vertical").

The flow vector for a pixel (x,y) is (Output[x,y,0], Output[x,y,1]).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_lmb-2Dfreiburg_flownet2_issues_77-23issuecomment-2D340165582&d=DwMFaQ&c=ZbgFmJjg4pdtrnL2HUJUDw&r=iPwEFBadX7fVtSOw208U6xhNNuaT4uFigfdW6CFf260&m=ebMpSNT9uc2QK5sLw8FSNJaq8ltHSNJPm4q5PBQ8xSI&s=RtXYN__OVEL2tUcXynpvrDbIgyCjcOC6y9KkOYdH7lQ&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AV1iUo930XvUFCOl82r1oQtHxbr3E0Orks5swvaXgaJpZM4QIEwk&d=DwMFaQ&c=ZbgFmJjg4pdtrnL2HUJUDw&r=iPwEFBadX7fVtSOw208U6xhNNuaT4uFigfdW6CFf260&m=ebMpSNT9uc2QK5sLw8FSNJaq8ltHSNJPm4q5PBQ8xSI&s=Bm99biH7eZ_0Jyx-QufjaaftV-ZEiFURz6QlLRdmJSQ&e=.

[nikolausmayer]

The Middlebury group has a code package for flow visualization: http://vision.middlebury.edu/flow/code/flow-code.zip or http://vision.middlebury.edu/flow/code/flow-code-matlab.zip

I'm not sure if that gives you the coding I used. My examples might not even be real flow visualizations, I just wanted to show that the object boundaries are indeed visible.

[Jingyao12]

I see, Thank you!

From: Nikolaus Mayer [mailto:notifications@github.com] Sent: Monday, October 30, 2017 3:57 AM To: lmb-freiburg/flownet2 Cc: Li, Jingyao; Author Subject: Re: [lmb-freiburg/flownet2] performance of Flownet2 is not good as the examples (#77)

The Middlebury group has a code package for flow visualization: http://vision.middlebury.edu/flow/code/flow-code.ziphttps://urldefense.proofpoint.com/v2/url?u=http-3A__vision.middlebury.edu_flow_code_flow-2Dcode.zip&d=DwMFaQ&c=ZbgFmJjg4pdtrnL2HUJUDw&r=iPwEFBadX7fVtSOw208U6xhNNuaT4uFigfdW6CFf260&m=sOYW28s6wdhIAYLPk2Ri-zpiM7nqRTmEFHxY2JDU1Zw&s=3j6zSundnXOEQULrLkE_QE4e1gQuniVQnFX44fxPr6A&e= or http://vision.middlebury.edu/flow/code/flow-code-matlab.ziphttps://urldefense.proofpoint.com/v2/url?u=http-3A__vision.middlebury.edu_flow_code_flow-2Dcode-2Dmatlab.zip&d=DwMFaQ&c=ZbgFmJjg4pdtrnL2HUJUDw&r=iPwEFBadX7fVtSOw208U6xhNNuaT4uFigfdW6CFf260&m=sOYW28s6wdhIAYLPk2Ri-zpiM7nqRTmEFHxY2JDU1Zw&s=3MmZGZ4AqU9qBSm3rgbKJ1aLJGTscle3tMmAylwxGdQ&e=

I'm not sure if that gives you the coding I used. My examples might not even be real flow visualizations, I just wanted to show that the object boundaries are indeed visible.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_lmb-2Dfreiburg_flownet2_issues_77-23issuecomment-2D340368568&d=DwMFaQ&c=ZbgFmJjg4pdtrnL2HUJUDw&r=iPwEFBadX7fVtSOw208U6xhNNuaT4uFigfdW6CFf260&m=sOYW28s6wdhIAYLPk2Ri-zpiM7nqRTmEFHxY2JDU1Zw&s=-kyKbx74X3k0UaHxzIubU8QNu5Zwc72lgmtaw_qOoxU&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AV1iUgiZ1jUVZLgS7ksoaiN0QqlKeWyhks5sxYFfgaJpZM4QIEwk&d=DwMFaQ&c=ZbgFmJjg4pdtrnL2HUJUDw&r=iPwEFBadX7fVtSOw208U6xhNNuaT4uFigfdW6CFf260&m=sOYW28s6wdhIAYLPk2Ri-zpiM7nqRTmEFHxY2JDU1Zw&s=VVzXvSZZ29fMdigRH1qglJRPCU23JVNGVNV-Ybo6jAc&e=.

[PkuRainBow]

This is the ground truth.

But below is the optical flow predicted by model "FlowNet2S" seems OK !

the optical flow predicted by model "FlowNetfCSS" seems strange !

The optical flow predicted by model "FlowNet2" is also strange !!

Very strange !!!

[nikolausmayer]

@PkuRainBow I'm getting good outputs for all those networks. Did you use our scripts to get these outputs?

[PkuRainBow]

I am using "run-flownet-many.py" and test the pretrained models you provide. My only concern is about the visualization code. I find one copy from webset like this:

################################ Reading flow file ################################
import numpy as np
import cv2

# flo_file = "D:\\users\\yuyua\\flow_net\\caffe\\data\\12.flo"
flow_name = "00003_flow.flo"
# flo_file = "X:\\Data\\FlyingChairs\\FlyingChairs_release\\data\\" + flow_name
flo_file = "D:\\users\\yuyua\\flow_net\\caffe\\data\\" + flow_name
f = open(flo_file, 'rb')

x = np.fromfile(f, np.int32, count=1) # not sure what this gives
w = np.fromfile(f, np.int32, count=1) # width
h = np.fromfile(f, np.int32, count=1) # height
print 'x %d, w %d, h %d flo file' % (x, w, h)

data = np.fromfile(f, np.float32) # vector

data_2D = np.reshape(data, newshape=(384,512,2)); # convert to x,y - flow
x = data_2D[...,0]; y = data_2D[...,1];

def draw_hsv(flow):
    h, w = flow.shape[:2]
    fx, fy = flow[:,:,0], flow[:,:,1]
    ang = np.arctan2(fy, fx) + np.pi
    v = np.sqrt(fx*fx+fy*fy)
    hsv = np.zeros((h, w, 3), np.uint8)
    hsv[...,0] = ang*(180/np.pi/2)
    hsv[...,1] = 255
    hsv[...,2] = np.minimum(v*4, 255)
    bgr = cv2.cvtColor(hsv, cv2.COLOR_HSV2BGR)
    return bgr

################################ visualising flow file ################################
mag, ang = cv2.cartToPolar(x,y)
hsv = np.zeros_like(x)
hsv = np.array([ hsv,hsv,hsv ])
hsv = np.reshape(hsv, (384,512,3)); # having rgb channel
hsv[...,1] = 255; # full green channel
hsv[...,0] = ang*180/np.pi/2 # angle in pi
hsv[...,2] = cv2.normalize(mag,None,0,255,cv2.NORM_MINMAX) # magnitude [0,255]
bgr = cv2.cvtColor(hsv,cv2.COLOR_HSV2BGR)
bgr = draw_hsv(data_2D)
cv2.imwrite(flow_name + '.png',bgr)

[PkuRainBow]

All the models are pre-trained by you. It seems very strange.