test.py运行报错

[imMid-Star]

Loading checkpoint: /1/dl_pzx/MASTER-pytorch/model_output/master_saved/models/MASTER_Default/example_0314_212753/model_best.pth with saved best metric 0.9623 test data size: 2 steps: 2 0%| | 0/2 [00:00<?, ?it/s]Fatal Python error: Cannot recover from stack overflow.

Current thread 0x00007fe5b3b2b3c0 (most recent call first): File "/home/dell10/anaconda3/envs/pzx/lib/python3.7/site-packages/PIL/_util.py", line 6 in isPath File "/home/dell10/anaconda3/envs/pzx/lib/python3.7/site-packages/PIL/Image.py", line 2971 in open File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 82 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem File "/1/dl_pzx/MASTER-pytorch/data_utils/datasets.py", line 104 in getitem ... 0%| | 0/2 [00:00<?, ?it/s] Traceback (most recent call last): File "/home/dell10/anaconda3/envs/pzx/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 990, in _try_get_data data = self._data_queue.get(timeout=timeout) File "/home/dell10/anaconda3/envs/pzx/lib/python3.7/multiprocessing/queues.py", line 104, in get if not self._poll(timeout): File "/home/dell10/anaconda3/envs/pzx/lib/python3.7/multiprocessing/connection.py", line 257, in poll return self._poll(timeout) File "/home/dell10/anaconda3/envs/pzx/lib/python3.7/multiprocessing/connection.py", line 414, in _poll r = wait([self], timeout) File "/home/dell10/anaconda3/envs/pzx/lib/python3.7/multiprocessing/connection.py", line 921, in wait ready = selector.select(timeout) File "/home/dell10/anaconda3/envs/pzx/lib/python3.7/selectors.py", line 415, in select fd_event_list = self._selector.poll(timeout) File "/home/dell10/anaconda3/envs/pzx/lib/python3.7/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler _error_if_any_worker_fails() RuntimeError: DataLoader worker (pid 31156) is killed by signal: Aborted.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/1/dl_pzx/MASTER-pytorch/test.py", line 130, in predict(args) File "/1/dl_pzx/MASTER-pytorch/test.py", line 61, in predict for step_idx, input_data_item in tqdm(enumerate(test_data_loader),total=len(test_data_loader)): File "/home/dell10/anaconda3/envs/pzx/lib/python3.7/site-packages/tqdm/std.py", line 1185, in iter for obj in iterable: File "/home/dell10/anaconda3/envs/pzx/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 521, in next data = self._next_data() File "/home/dell10/anaconda3/envs/pzx/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1186, in _next_data idx, data = self._get_data() File "/home/dell10/anaconda3/envs/pzx/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1152, in _get_data success, data = self._try_get_data() File "/home/dell10/anaconda3/envs/pzx/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1003, in _try_get_data raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly error is {}'.format(pids_str,e)) from e RuntimeError: DataLoader worker (pid(s) 31156) exited unexpectedly error is DataLoader worker (pid 31156) is killed by signal: Aborted.

bs设置为了1，num_worker也为1，DataLoader报错的pid也一直在变化，请问是什么原因？