search

zigvu / khajuri

Video Pipeline
0 stars 0 forks source link

Error with dump_patches_for_curation from longer video #81

Closed arpgh closed 9 years ago

arpgh commented 9 years ago

Branch: Development Video: wc14-NetCos-xtraTm.mkv Path: gpu2:/mnt/data/wc14cls/training/0seed-cleaned/vdo-test/wc14-NetCos-xtraTm/ Log with error: runlog-dump_patches_for_curation.txt Json folder used: json Error: 

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 302, in _handle_workers
    pool._maintain_pool()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 206, in _maintain_pool
    self._repopulate_pool()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 199, in _repopulate_pool
    w.start()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 130, in start
    self._popen = Popen(self)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 120, in __init__
    self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
eacharya commented 9 years ago

The out of memory issue was caused by adding a lot of processes to the Pool while parallelizing dump patches in this commit:

https://github.com/zigvu/khajuri/commit/0de663aeef94dc54192e4a033dae4911cf649bb4#diff-16bccffe8aa9b98d971800344bb8a088

Following changes were made so that dump patches would be fast but not cause out-of-memory issues:

  1. Parallelize JSON reading and curation extraction in CurationManager.py
  2. Create multiple VideoReader.py objects to save frames in parallel and extract patches from them
  3. Encapsulate dump logic in a class file - leave the bin script light weight
eacharya commented 9 years ago

Need also to parallelize CurationManager.aggregateCurationPatches() for the curation to be parallelized throughout.

eacharya commented 9 years ago

Prallelized. Merged to development.

Note: Previously, we were re-writing self.curationBboxes[classId] data structure after aggregateCurationPatches. We no longer do that anymore because of parallelization. While that data structure is only used by aggregateCurationPatches and only used once, in the future, care should be taken if modifying that global data structure. In such a case, instead of passing global data, pass in through function calls.