joheras / CLoDSA

122 stars 33 forks source link

problem in instance-segmentation #82

Open kaphleamrit2 opened 1 year ago

kaphleamrit2 commented 1 year ago

I have 53 RGB images and their corresponding annotation in coco JSON format. I used the colab notebook (instance-segmentation) from the example and just replaced datasets with mine. It starts the processes, generated few images andgives the following error. I can't figure it out. Please help. I am running it on google colab pro.

`/usr/local/lib/python3.8/dist-packages/joblib/externals/loky/process_executor.py:700: UserWarning: A worker stopped while some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak.
  warnings.warn(
ERROR:concurrent.futures:exception calling callback for <Future at 0x7f40abc1a640 state=finished raised TerminatedWorkerError>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/joblib/externals/loky/_base.py", line 26, in _invoke_callbacks
    callback(self)
  File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 385, in __call__
    self.parallel.dispatch_next()
  File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 834, in dispatch_next
    if not self.dispatch_one_batch(self._original_iterator):
  File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 901, in dispatch_one_batch
    self._dispatch(tasks)
  File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 819, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
  File "/usr/local/lib/python3.8/dist-packages/joblib/_parallel_backends.py", line 556, in apply_async
    future = self._workers.submit(SafeFunction(func))
  File "/usr/local/lib/python3.8/dist-packages/joblib/externals/loky/reusable_executor.py", line 176, in submit
    return super().submit(fn, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/joblib/externals/loky/process_executor.py", line 1129, in submit
    raise self._flags.broken
joblib.externals.loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.

The exit codes of the workers are {SIGSEGV(-11)}
---------------------------------------------------------------------------
TerminatedWorkerError                     Traceback (most recent call last)
[<ipython-input-39-1d9c8eaf67b6>](https://localhost:8080/#) in <module>
----> 1 augmentor.applyAugmentation()

13 frames
[/usr/local/lib/python3.8/dist-packages/joblib/externals/loky/process_executor.py](https://localhost:8080/#) in submit(self, fn, *args, **kwargs)
   1127         with self._flags.shutdown_lock:
   1128             if self._flags.broken is not None:
-> 1129                 raise self._flags.broken
   1130             if self._flags.shutdown:
   1131                 raise ShutdownExecutorError(

TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.

The exit codes of the workers are {SIGSEGV(-11)}`
rjbray915 commented 1 year ago

I'm running into the same problem. Even with 32gb of ram, I can only run augmentations on 2 images at a time. Seems to use lots of memory depending on how many annotations there are.