search

facebookresearch / maskrcnn-benchmark

Fast, modular reference implementation of Instance Segmentation and Object Detection algorithms in PyTorch.
MIT License
9.3k stars 2.49k forks source link

RuntimeError: DataLoader worker (pid 16560) is killed by signal: Killed. #195

Open jario-jin opened 5 years ago

jario-jin commented 5 years ago

❓ Questions and Help

I use COCO dataset and default setting. This error occurred when running to 23540 iters. Is this my dataset problem, or?

My Environment: 2018-11-21 17:02:50,065 maskrcnn_benchmark INFO: Using 2 GPUs 2018-11-21 17:02:50,066 maskrcnn_benchmark INFO: Namespace(config_file='configs/e2e_faster_rcnn_R_50_FPN_1x.yaml', distributed=True, local_rank=0, opts=[], skip_test=False) 2018-11-21 17:02:50,066 maskrcnn_benchmark INFO: Collecting env info (might take some time) 2018-11-21 17:02:51,106 maskrcnn_benchmark INFO: PyTorch version: 1.0.0a0+5d0ef34 Is debug build: No CUDA used to build PyTorch: 9.1.85

OS: Ubuntu 16.04.4 LTS GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609 CMake version: version 3.5.1

Python version: 2.7 Is CUDA available: Yes CUDA runtime version: Could not collect GPU models and configuration: GPU 0: GeForce GTX 1080 Ti GPU 1: TITAN X (Pascal)

Nvidia driver version: 390.59 cuDNN version: Probably one of the following: /usr/local/cuda-9.1/lib64/libcudnn.so /usr/local/cuda-9.1/lib64/libcudnn.so.7 /usr/local/cuda-9.1/lib64/libcudnn.so.7.0.5 /usr/local/cuda-9.1/lib64/libcudnn_static.a

Error Info: 2018-11-21 19:57:51,134 maskrcnn_benchmark.trainer INFO: eta: 1 day, 17:36:49 iter: 23540 loss: 0.6437 (0.6677) loss_box_reg: 0.1438 (0.1584) loss_rpn_box_reg: 0.0716 (0.0752) loss_classifier: 0.3284 (0.3506) time: 0.4422 (0.4453) loss_objectness: 0.0696 (0.0836) data: 0.0059 (0.0088) lr: 0.005000 max mem: 3546 2018-11-21 19:57:59,999 maskrcnn_benchmark.trainer INFO: eta: 1 day, 17:36:39 iter: 23560 loss: 0.5838 (0.6677) loss_box_reg: 0.1453 (0.1583) loss_rpn_box_reg: 0.0342 (0.0751) loss_classifier: 0.3052 (0.3506) time: 0.4419 (0.4452) loss_objectness: 0.0557 (0.0836) data: 0.0045 (0.0088) lr: 0.005000 max mem: 3546 2018-11-21 19:58:08,802 maskrcnn_benchmark.trainer INFO: eta: 1 day, 17:36:29 iter: 23580 loss: 0.6262 (0.6676) loss_box_reg: 0.1667 (0.1583) loss_rpn_box_reg: 0.0962 (0.0752) loss_classifier: 0.2801 (0.3505) time: 0.4419 (0.4452) loss_objectness: 0.0805 (0.0836) data: 0.0054 (0.0088) lr: 0.005000 max mem: 3546 2018-11-21 19:58:17,906 maskrcnn_benchmark.trainer INFO: eta: 1 day, 17:36:23 iter: 23600 loss: 0.5771 (0.6676) loss_box_reg: 0.1431 (0.1583) loss_rpn_box_reg: 0.0493 (0.0752) loss_classifier: 0.3077 (0.3505) time: 0.4511 (0.4453) loss_objectness: 0.0478 (0.0836) data: 0.0049 (0.0088) lr: 0.005000 max mem: 3546 2018-11-21 19:58:26,721 maskrcnn_benchmark.trainer INFO: eta: 1 day, 17:36:13 iter: 23620 loss: 0.6393 (0.6676) loss_box_reg: 0.1706 (0.1583) loss_rpn_box_reg: 0.0479 (0.0752) loss_classifier: 0.3366 (0.3505) time: 0.4390 (0.4452) loss_objectness: 0.0615 (0.0836) data: 0.0050 (0.0088) lr: 0.005000 max mem: 3546 Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/queues.py", line 268, in _feed send(obj) File "/usr/local/lib/python2.7/dist-packages/torch/multiprocessing/queue.py", line 18, in send self.send_bytes(buf.getvalue()) IOError: [Errno 32] Broken pipe Traceback (most recent call last): File "tools/train_net.py", line 170, in main() File "tools/train_net.py", line 163, in main model = train(cfg, args.local_rank, args.distributed) File "tools/train_net.py", line 73, in train arguments, File "/home/jario/spire-net-1810/maskrcnn-benchmark/maskrcnn_benchmark/engine/trainer.py", line 66, in do_train loss_dict = model(images, targets) File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 479, in call result = self.forward(*input, kwargs) File "/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/deprecated/distributed.py", line 223, in forward return self.module(*inputs[0], *kwargs[0]) File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 479, in call result = self.forward(input, kwargs) File "/home/jario/spire-net-1810/maskrcnn-benchmark/maskrcnn_benchmark/modeling/detector/generalized_rcnn.py", line 49, in forward features = self.backbone(images.tensors) File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 479, in call result = self.forward(*input, kwargs) File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/container.py", line 92, in forward input = module(input) File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 479, in call result = self.forward(*input, *kwargs) File "/home/jario/spire-net-1810/maskrcnn-benchmark/maskrcnn_benchmark/modeling/backbone/resnet.py", line 117, in forward x = self.stem(x) File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 479, in call result = self.forward(input, kwargs) File "/home/jario/spire-net-1810/maskrcnn-benchmark/maskrcnn_benchmark/modeling/backbone/resnet.py", line 287, in forward x = self.conv1(x) File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 479, in call result = self.forward(*input, **kwargs) File "/home/jario/spire-net-1810/maskrcnn-benchmark/maskrcnn_benchmark/layers/misc.py", line 33, in forward return super(Conv2d, self).forward(x) File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/conv.py", line 313, in forward self.padding, self.dilation, self.groups) File "/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.py", line 274, in handler _error_if_any_worker_fails() RuntimeError: DataLoader worker (pid 16560) is killed by signal: Killed.

jario-jin commented 5 years ago

I used Detectron (https://github.com/facebookresearch/Detectron) for training in the same environment and did not encounter this problem. This means that the dataset should be all right. Anyone have the same problem?

fmassa commented 5 years ago

Can you try setting DATALOADER.NUM_WORKERS to 0 and run again? This should give you a better idea of the problem.

Also, one possibility is that one of the images of your dataset is corrupted, but because opencv (which is used by Detectron) loads corrupted images without complaining (while PIL by default raises a warning), this might be the issue.

LaoYang1994 commented 5 years ago

I have run into this problem twice. Does anyone have solved this problem?

jario-jin commented 5 years ago

This problem occasionally occurs. When I set DATALOADER. NUM_WORKERS to 0, this error did not occur again, but the training speed would decrease. I still haven't found the specific reason.

LaoYang1994 commented 5 years ago

Except setting DATALOADER. NUM_WORKERS to 0, is there any other ways to solve this problem?

eric-xw commented 5 years ago

I encountered the same issue at iteration 54700. Any fix for this?

eric-xw commented 5 years ago

In my case, it was caused by the CPU memory. After I increased my CPU memory on the cluster, the problem is gone.

lironmo commented 5 years ago

same here: ERROR - worker:73 - call - trainer crashed. Error: DataLoader worker (pid 1159) is killed by signal: Killed.

mainguyenanhvu commented 5 years ago

I have run it in CPU and also met an error: RuntimeError: DataLoader worker (pid 16560) is killed by signal: Killed.

I followed @fmassa, it was killed and printed in terminal:

Count of instances per bin: [85308 31958] Test: [ 0/5000] eta: 8:26:32 model_time: 5.8156 (5.8156) evaluator_time: 0.1168 (0.1168) time: 6.0785 data: 0.1460 Killed

Please, help me. Thanks a lot.

SaynaEbrahimi commented 5 years ago

I removed def _init_fn(): np.random.seed(args.seed) form my torch.utils.data.DataLoader(). Now it is working with 4 workers but it is still not using multiple gpus when I use torch.nn.DataParallel(model).cuda(). Does anyone encountered this?

Tato14 commented 4 years ago

Hi @SaynaEbrahimi, I am not able to find def _init_fn(): np.random.seed(args.seed) in the dataloader.py file. Should I find it somewhere else? Or is a matter of versions?

qingnanli commented 4 years ago

@SaynaEbrahimi @Tato14 Have you solved this problem? I met the same problem...https://github.com/xingyizhou/CenterNet/issues/566

b5l commented 4 years ago

If you are also experiencing this problem, I found a solution: In my case, the Out Of Memory Killer killed the process. The solution was to create a bigger swap file and let the system use that:

linux@linux:~# fallocate -l 64G /tmp/swap
linux@linux:~# chmod 600 /tmp/swap
linux@linux:~# mkswap /tmp/swap
linux@linux:~# swapon /tmp/swap

You can verify that the swap is working by running swapon --show.

Michel-liu commented 4 years ago

For me, this problem sometimes happens in a docker virtual machine. Commonly, I will set the cpu count lager than NUM_WORKERS & use a bigger memory than the dataset used.

lishen commented 4 years ago

If you are also experiencing this problem, I found a solution: In my case, the Out Of Memory Killer killed the process. The solution was to create a bigger swap file and let the system use that:

linux@linux:~# fallocate -l 64G /tmp/swap
linux@linux:~# chmod 600 /tmp/swap
linux@linux:~# mkswap /tmp/swap
linux@linux:~# swapon /tmp/swap

You can verify that the swap is working by running swapon --show.

Yeah! This solved my problem. I can finally use multiple workers.

pzhren commented 2 years ago

For me, this problem sometimes happens in a docker virtual machine. Commonly, I will set the cpu count lager than NUM_WORKERS & use a bigger memory than the dataset used. I'm in the same situation, how do I fix it?