search

lmb-freiburg / flownet2

FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks
https://lmb.informatik.uni-freiburg.de/Publications/2017/IMKDB17/
Other
1k stars 318 forks source link

Cannot run run-flownet-many for 2 or more image pairs #72

Closed letatanu closed 7 years ago

letatanu commented 7 years ago

When I run like one by one pair, it works very well. However, when I tried to run 2 or more pairs at the same time, the first pair is good, but there's an error on the second pair. F1010 17:50:35.829269 5211 cudnn_conv_layer.cpp:53] Check failed: status == CUDNN_STATUS_SUCCESS (4 vs. 0) CUDNN_STATUS_INTERNAL_ERROR *** Check failure stack trace: *** Aborted (core dumped) Anyone gets this problem?

nikolausmayer commented 7 years ago

Does this help? https://github.com/lmb-freiburg/flownet2/issues/22 This issue is more about memory exhaustion, but it also references input images of non-uniform size (which does not work).

letatanu commented 7 years ago

Yeah. This problem is about memory exhaustion. My input images are in uniform size because I extract them from the same video. I did a test. I ran run-flownet-many 1 pair each time for many times like 200 times. It worked very well on around the first 100 pairs. However, the memory problem still showed after that. Thus, I think after using run-flownet-many, caffe didn't release memory. Then, when I run run-flownet-many again, caffe declared another memory. This led to memory exhaustion after using many times run-flownet-many. I'm still looking for a solution.

nikolausmayer commented 7 years ago

Do the followup comments in https://github.com/lmb-freiburg/flownet2/issues/22 help?

letatanu commented 7 years ago

It doesn't help me. My solution is adding del net at the end. Now it's working perfectly.

nikolausmayer commented 6 years ago

Hm, that's a pretty brutal solution. But if it solves the problem, I won't complain... :)