search

myfavouritekk / T-CNN

ImageNet 2015 Object Detection from Video (VID)
MIT License
369 stars 152 forks source link

matlab.engine.EngineError: MATLAB function cannot be evaluated #5

Open learntolearn11 opened 8 years ago

learntolearn11 commented 8 years ago

[2016-07-26 20:49:44,869 24261 greedy_tracking_from_raw_dets.py:58 INFO] Tracking lion... [2016-07-26 20:49:45,243 24261 track.py:219 INFO] Upon low confidence: total 0 tracks [2016-07-26 20:49:45,244 24261 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class lion [2016-07-26 20:49:45,244 24261 greedy_tracking_from_raw_dets.py:58 INFO] Tracking motorcycle... [2016-07-26 20:49:45,617 24261 track.py:219 INFO] Upon low confidence: total 0 tracks [2016-07-26 20:49:45,618 24261 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class motorcycle [2016-07-26 20:49:45,618 24261 greedy_tracking_from_raw_dets.py:58 INFO] Tracking cattle... [2016-07-26 20:49:45,990 24261 track.py:219 INFO] Upon low confidence: total 0 tracks [2016-07-26 20:49:45,991 24261 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class cattle [2016-07-26 20:49:45,991 24261 greedy_tracking_from_raw_dets.py:58 INFO] Tracking airplane... [2016-07-26 20:49:46,336 24261 track.py:222 INFO] tracking top No.0 in ILSVRC2015_val_00007011 [2016-07-26 20:49:46,336 24261 common.py:341 DEBUG] Use opened Matlab session: <matlab.engine.matlabengine.MatlabEngine object at 0x7f1e308fbcd0> [2016-07-26 20:50:08,061 24261 common.py:341 DEBUG] Use opened Matlab session: <matlab.engine.matlabengine.MatlabEngine object at 0x7f1e308fbcd0> [2016-07-26 20:50:09,544 24261 common.py:348 ERROR] Existing engine no response: <matlab.engine.matlabengine.MatlabEngine object at 0x7f1e308fbcd0> [2016-07-26 20:50:13,866 24261 common.py:341 DEBUG] Use opened Matlab session: <matlab.engine.matlabengine.MatlabEngine object at 0x7f1e711e4710> [2016-07-26 20:50:29,874 24261 common.py:348 ERROR] Existing engine no response: <matlab.engine.matlabengine.MatlabEngine object at 0x7f1e711e4710> Traceback (most recent call last): File "tools/tracking/greedy_tracking_from_raw_dets.py", line 61, in cls_idx, opts) File "./vdetlib/vdet/track.py", line 234, in greedily_track_from_raw_dets new_tracks = track_method(vid_proto, anchor_frame_id, anchor_bbox, opts) File "./vdetlib/vdet/track.py", line 82, in fcn_tracker [matlab.double(bbox),] + fw_frames + [opts.gpu,], opts.engine) File "./vdetlib/utils/common.py", line 349, in matlab_engine raise e matlab.engine.EngineError: MATLAB function cannot be evaluated Error in atexit._run_exitfuncs: Traceback (most recent call last): File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs func(_targs, _kargs) File "/home/tju/.local/lib/python2.7/site-packages/matlab/engine/init.py", line 186, in exit_engines eng().exit() File "/home/tju/.local/lib/python2.7/site-packages/matlab/engine/matlabengine.py", line 246, in exit pythonengine.closeMATLAB(self.__dict["_matlab"]) SystemError: MATLAB process cannot be terminated. Error in sys.exitfunc: Traceback (most recent call last): File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs func(_targs, _kargs) File "/home/tju/.local/lib/python2.7/site-packages/matlab/engine/init.py", line 186, in exit_engines eng().exit() File "/home/tju/.local/lib/python2.7/site-packages/matlab/engine/matlabengine.py", line 246, in exit pythonengine.closeMATLAB(self.__dict["_matlab"]) SystemError: MATLAB process cannot be terminated. Exception SystemError: 'MATLAB process cannot be terminated.' in <bound method MatlabEngine.del of <matlab.engine.matlabengine.MatlabEngine object at 0x7f1e711e4710>> ignored

doses anyone happen to this? please let me know your solution, Thanks!

zawlin commented 8 years ago

try adding caffe.reset_all(); to fcn_tracker.m just before caffe.set_mode_gpu();. at least for me the memory keep running out because each call of the script recreate the entire network and the matlab engine keep crashing, that's why you see this error.

myfavouritekk commented 8 years ago

@zawlin Right. reset_all() will clear stand-alone nets in the GPU. I think adding reset_all() for each fcn_tracker call may solve this problem, but it requires loading VGG for each tracking process and may takes a lot of time compared to tracking itself. Probably should call rest_all only when restarting a matlab engine. What do you think?

zong209 commented 7 years ago

I got the same error. I try to add caffe.reset_all; in fcn_tracker.m, but the same crash happened. I try to add it in track.py by matlab_energy(), but I am failed. Now I don‘t know how to solve it.

[2016-10-12 11:26:06,145 10341 greedy_tracking_from_raw_dets.py:42 INFO] Video: ILSVRC2015_val_00007011
[2016-10-12 11:26:09,939 10341 greedy_tracking_from_raw_dets.py:58 INFO] Tracking turtle...
[2016-10-12 11:26:10,188 10341 track.py:219 INFO] Upon low confidence: total 0 tracks
[2016-10-12 11:26:10,190 10341 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class turtle
[2016-10-12 11:26:10,190 10341 greedy_tracking_from_raw_dets.py:58 INFO] Tracking antelope...
[2016-10-12 11:26:10,421 10341 track.py:219 INFO] Upon low confidence: total 0 tracks
[2016-10-12 11:26:10,422 10341 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class antelope
[2016-10-12 11:26:10,422 10341 greedy_tracking_from_raw_dets.py:58 INFO] Tracking bicycle...
[2016-10-12 11:26:10,659 10341 track.py:219 INFO] Upon low confidence: total 0 tracks
[2016-10-12 11:26:10,660 10341 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class bicycle
[2016-10-12 11:26:10,660 10341 greedy_tracking_from_raw_dets.py:58 INFO] Tracking lion...
[2016-10-12 11:26:10,894 10341 track.py:219 INFO] Upon low confidence: total 0 tracks
[2016-10-12 11:26:10,895 10341 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class lion
[2016-10-12 11:26:10,895 10341 greedy_tracking_from_raw_dets.py:58 INFO] Tracking motorcycle...
[2016-10-12 11:26:11,127 10341 track.py:219 INFO] Upon low confidence: total 0 tracks
[2016-10-12 11:26:11,129 10341 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class motorcycle
[2016-10-12 11:26:11,129 10341 greedy_tracking_from_raw_dets.py:58 INFO] Tracking cattle...
[2016-10-12 11:26:11,363 10341 track.py:219 INFO] Upon low confidence: total 0 tracks
[2016-10-12 11:26:11,365 10341 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class cattle
[2016-10-12 11:26:11,365 10341 greedy_tracking_from_raw_dets.py:58 INFO] Tracking airplane...
[2016-10-12 11:26:11,560 10341 track.py:222 INFO] tracking top No.0 in ILSVRC2015_val_00007011
[2016-10-12 11:26:11,561 10341 common.py:341 DEBUG] Use opened Matlab session: <matlab.engine.matlabengine.MatlabEngine object at 0x7f5fe9562d10>
[2016-10-12 11:26:14,244 10341 common.py:348 ERROR] Existing engine no response: <matlab.engine.matlabengine.MatlabEngine object at 0x7f5fe9562d10>
[2016-10-12 11:26:17,436 10341 common.py:341 DEBUG] Use opened Matlab session: <matlab.engine.matlabengine.MatlabEngine object at 0x7f5fe95a8550>
[2016-10-12 11:26:20,180 10341 common.py:348 ERROR] Existing engine no response: <matlab.engine.matlabengine.MatlabEngine object at 0x7f5fe95a8550>
Traceback (most recent call last):
  File "tools/tracking/greedy_tracking_from_raw_dets.py", line 61, in <module>
    cls_idx, opts)
  File "./vdetlib/vdet/track.py", line 234, in greedily_track_from_raw_dets
    new_tracks = track_method(vid_proto, anchor_frame_id, anchor_bbox, opts)
  File "./vdetlib/vdet/track.py", line 82, in fcn_tracker
    [matlab.double(bbox),] + fw_frames + [opts.gpu,], opts.engine)
  File "./vdetlib/utils/common.py", line 349, in matlab_engine
    raise e
matlab.engine.EngineError: MATLAB function cannot be evaluated
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/local/lib/python2.7/dist-packages/matlab/engine/__init__.py", line 186, in __exit_engines
    eng().exit()
  File "/usr/local/lib/python2.7/dist-packages/matlab/engine/matlabengine.py", line 246, in exit
    pythonengine.closeMATLAB(self.__dict__["_matlab"])
SystemError: MATLAB process cannot be terminated.
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/local/lib/python2.7/dist-packages/matlab/engine/__init__.py", line 186, in __exit_engines
    eng().exit()
  File "/usr/local/lib/python2.7/dist-packages/matlab/engine/matlabengine.py", line 246, in exit
    pythonengine.closeMATLAB(self.__dict__["_matlab"])
SystemError: MATLAB process cannot be terminated.
Exception SystemError: 'MATLAB process cannot be terminated.' in <bound method MatlabEngine.__del__ of <matlab.engine.matlabengine.MatlabEngine object at 0x7f5fe95a8550>> ignored

My GPU is GTX960M with 2GB, is it the reason? Does it mean that I should use an another caffemodel ？

myfavouritekk commented 7 years ago

@zong209 what is matlab_energy?

zong209 commented 7 years ago

I'm sorry, I didn't express it clearly @myfavouritekk . I actually try to add reset_all(); before matlab.engine.start. For above problem, I changed my GPU into GTX1080, then added caffe.reset_all(); before caffe.set_mode_gpu(); in fcn_tracker.m. The crash didn't happen again.

Jensen-Su commented 7 years ago

I've tried adding caffe.reset_all(); before caffe.set_mode_gpu(); in fcn_tracker.m right as what @zawlin @myfavouritekk and @zong209 suggested. And my GPU is TITAN X with 12GB. But nothing changed, why?

Cleared 0 solvers and 0 stand-alone nets
[2016-12-29 19:08:59,996 8369 greedy_tracking_from_raw_dets.py:42 INFO] Video: ILSVRC2015_val_00007011
[2016-12-29 19:09:03,220 8369 greedy_tracking_from_raw_dets.py:58 INFO] Tracking turtle...
[2016-12-29 19:09:03,597 8369 track.py:219 INFO] Upon low confidence: total 0 tracks
[2016-12-29 19:09:03,599 8369 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class turtle
[2016-12-29 19:09:03,599 8369 greedy_tracking_from_raw_dets.py:58 INFO] Tracking antelope...
[2016-12-29 19:09:03,978 8369 track.py:219 INFO] Upon low confidence: total 0 tracks
[2016-12-29 19:09:03,980 8369 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class antelope
[2016-12-29 19:09:03,980 8369 greedy_tracking_from_raw_dets.py:58 INFO] Tracking bicycle...
[2016-12-29 19:09:04,359 8369 track.py:219 INFO] Upon low confidence: total 0 tracks
[2016-12-29 19:09:04,361 8369 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class bicycle
[2016-12-29 19:09:04,361 8369 greedy_tracking_from_raw_dets.py:58 INFO] Tracking lion...
[2016-12-29 19:09:04,748 8369 track.py:219 INFO] Upon low confidence: total 0 tracks
[2016-12-29 19:09:04,750 8369 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class lion
[2016-12-29 19:09:04,750 8369 greedy_tracking_from_raw_dets.py:58 INFO] Tracking motorcycle...
[2016-12-29 19:09:05,135 8369 track.py:219 INFO] Upon low confidence: total 0 tracks
[2016-12-29 19:09:05,136 8369 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class motorcycle
[2016-12-29 19:09:05,136 8369 greedy_tracking_from_raw_dets.py:58 INFO] Tracking cattle...
[2016-12-29 19:09:05,520 8369 track.py:219 INFO] Upon low confidence: total 0 tracks
[2016-12-29 19:09:05,522 8369 greedy_tracking_from_raw_dets.py:63 WARNING] ILSVRC2015_val_00007011 has no tracks for class cattle
[2016-12-29 19:09:05,522 8369 greedy_tracking_from_raw_dets.py:58 INFO] Tracking airplane...
[2016-12-29 19:09:05,867 8369 track.py:222 INFO] tracking top No.0 in ILSVRC2015_val_00007011
[2016-12-29 19:09:05,868 8369 common.py:341 DEBUG] Use opened Matlab session: <matlab.engine.matlabengine.MatlabEngine object at 0x7f560e086190>
[2016-12-29 19:09:22,661 8369 common.py:341 DEBUG] Use opened Matlab session: <matlab.engine.matlabengine.MatlabEngine object at 0x7f5632c66ed0>
Traceback (most recent call last):
  File "tools/tracking/greedy_tracking_from_raw_dets.py", line 61, in <module>
    cls_idx, opts)
  File "./vdetlib/vdet/track.py", line 234, in greedily_track_from_raw_dets
    new_tracks = track_method(vid_proto, anchor_frame_id, anchor_bbox, opts)
  File "./vdetlib/vdet/track.py", line 82, in fcn_tracker
    [matlab.double(bbox),] + fw_frames + [opts.gpu,], opts.engine)
  File "./vdetlib/utils/common.py", line 354, in matlab_engine
    eng.clear(nargout=0)
  File "/usr/local/lib/python2.7/dist-packages/matlab/engine/matlabengine.py", line 84, in __call__
    _stderr).result()
  File "/usr/local/lib/python2.7/dist-packages/matlab/engine/futureresult.py", line 106, in result
    self._future,self._nargout, None, out=self._out, err=self._err)
matlab.engine.EngineError: MATLAB function cannot be evaluated
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/local/lib/python2.7/dist-packages/matlab/engine/__init__.py", line 158, in __exit_engines
    eng().exit()
  File "/usr/local/lib/python2.7/dist-packages/matlab/engine/matlabengine.py", line 258, in exit
    pythonengine.closeMATLAB(self.__dict__["_matlab"])
SystemError: MATLAB process cannot be terminated.
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/local/lib/python2.7/dist-packages/matlab/engine/__init__.py", line 158, in __exit_engines
    eng().exit()
  File "/usr/local/lib/python2.7/dist-packages/matlab/engine/matlabengine.py", line 258, in exit
    pythonengine.closeMATLAB(self.__dict__["_matlab"])
SystemError: MATLAB process cannot be terminated.
Exception SystemError: 'MATLAB process cannot be terminated.' in <bound method MatlabEngine.__del__ of <matlab.engine.matlabengine.MatlabEngine object at 0x7f5632c66ed0>> ignored
mingde510 commented 7 years ago

I added caffe.reset_all(); before caffe.set_mode_gpu(); in fcn_tracker.m, the problem had not solve, Can you help me?@myfavouritekk

EswarSaiKrish commented 6 years ago

@mingde510 or anyone else, were you able to solve this issue finally ? I am using a GTX 1060 6 GB Variant and I get the same issue as this thread, MATLAB process cannot be terminated.

I have tried adding caffe.reset_all(); in fcn_tracker.m just before caffe.set_mode_gpu(); like @myfavouritekk, @zong209 has suggested, but my terminal output says Cleared 0 solvers and 0 stand-alone nets And then I get the same error.

Anyone has any success?? Thanks.

qiumei1101 commented 4 years ago

so who solve this problem now, thanks?