There is an error when i traiing.

[chaoshengzhe]

/usr/bin/python /media/data/liuben/CornerNet-Lite/train.py Process 0: loading all datasets... Process 0: using 4 workers loading annotations into memory... Done (t=0.00s) creating index... index created! loading annotations into memory... Done (t=0.00s) creating index... index created! loading annotations into memory... Done (t=0.00s) creating index... index created! loading annotations into memory... Done (t=0.00s) creating index... index created! loading annotations into memory... Done (t=0.00s) creating index... index created! system config... {'batch_size': 48, 'cache_dir': './cache', 'chunk_sizes': [12, 12, 12, 12], 'config_dir': './config', 'data_dir': './data', 'data_rng': <mtrand.RandomState object at 0x7fc160b361f8>, 'dataset': 'COCO', 'decay_rate': 10, 'display': 5, 'learning_rate': 0.00025, 'max_iter': 500000, 'nnet_rng': <mtrand.RandomState object at 0x7fc160b36240>, 'opt_algo': 'adam', 'prefetch_size': 5, 'pretrain': None, 'result_dir': './results', 'sampling_function': 'cornernet_saccade', 'snapshot': 5000, 'snapshot_name': 'CornerNet_Saccade', 'stepsize': 450000, 'test_split': 'testdev', 'train_split': 'trainval', 'val_iter': 100, 'val_split': 'minival'} db config... {'ae_threshold': 0.3, 'att_max_crops': 30, 'att_nms_ks': [3, 3, 3], 'att_ranges': [[96, 256], [32, 96], [0, 32]], 'att_ratios': [16, 8, 4], 'att_scales': [[1, 2, 4]], 'att_sizes': [[16, 16], [32, 32], [64, 64]], 'att_thresholds': [0.3], 'border': 64, 'categories': 2, 'data_aug': True, 'gaussian_bump': True, 'gaussian_iou': 0.5, 'gaussian_radius': -1, 'init_sizes': [192, 255], 'input_size': [255, 255], 'lighting': True, 'max_per_image': 100, 'max_per_set': 40, 'max_scale': 32, 'merge_bbox': False, 'min_scale': 16, 'nms_algorithm': 'exp_soft_nms', 'nms_kernel': 3, 'nms_threshold': 0.5, 'num_dets': 12, 'output_sizes': [[64, 64]], 'rand_center': True, 'rand_color': False, 'rand_crop': False, 'rand_scale_max': 1.1, 'rand_scale_min': 0.5, 'rand_scale_step': 0.1, 'rand_scales': array([0.5, 0.6, 0.7, 0.8, 0.9, 1. , 1.1]), 'ref_dets': True, 'score_threshold': 0.05, 'test_flipped': True, 'test_scales': [1], 'top_k': 12, 'view_sizes': [], 'weight_exp': 8} len of db: 25 distributed: False Process 0: building model... total parameters: 116969339 start prefetching data... shuffling indices... start prefetching data... shuffling indices... start prefetching data... shuffling indices... start prefetching data... shuffling indices... start prefetching data... setting learning rate to: 0.00025 training start... shuffling indices... 0%| | 0/500000 [00:00<?, ?it/s]shuffling indices... shuffling indices... shuffling indices... shuffling indices... shuffling indices... shuffling indices... shuffling indices... shuffling indices... shuffling indices... shuffling indices... shuffling indices... shuffling indices... shuffling indices... shuffling indices... shuffling indices... shuffling indices... shuffling indices...

shuffling indices... Traceback (most recent call last): File "/media/data/liuben/CornerNet-Lite/train.py", line 250, in main(None, ngpus_per_node, args) File "/media/data/liuben/CornerNet-Lite/train.py", line 234, in main train(training_dbs, validation_db, system_config, model, args) File "/media/data/liuben/CornerNet-Lite/train.py", line 166, in train training_loss = nnet.train(*training) File "/media/data/liuben/CornerNet-Lite/core/nnet/py_factory.py", line 93, in train loss = self.network(xs, ys) File "/home/teamway/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(input, *kwargs) File "/media/data/liuben/CornerNet-Lite/core/models/py_utils/data_parallel.py", line 66, in forward inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids, self.chunk_sizes) File "/media/data/liuben/CornerNet-Lite/core/models/py_utils/data_parallel.py", line 77, in scatter return scatter_kwargs(inputs, kwargs, device_ids, dim=self.dim, chunk_sizes=self.chunk_sizes) File "/media/data/liuben/CornerNet-Lite/core/models/py_utils/scatter_gather.py", line 30, in scatter_kwargs inputs = scatter(inputs, target_gpus, dim, chunk_sizes) if inputs else [] File "/media/data/liuben/CornerNet-Lite/core/models/py_utils/scatter_gather.py", line 25, in scatter return scatter_map(inputs) File "/media/data/liuben/CornerNet-Lite/core/models/py_utils/scatter_gather.py", line 18, in scatter_map return list(zip(map(scatter_map, obj))) File "/media/data/liuben/CornerNet-Lite/core/models/py_utils/scatter_gather.py", line 20, in scatter_map return list(map(list, zip(map(scatter_map, obj)))) File "/media/data/liuben/CornerNet-Lite/core/models/py_utils/scatter_gather.py", line 15, in scatter_map return Scatter.apply(target_gpus, chunk_sizes, dim, obj) File "/home/teamway/.local/lib/python3.6/site-packages/torch/nn/parallel/_functions.py", line 89, in forward outputs = comm.scatter(input, target_gpus, chunk_sizes, ctx.dim, streams) File "/home/teamway/.local/lib/python3.6/site-packages/torch/cuda/comm.py", line 147, in scatter return tuple(torch._C._scatter(tensor, devices, chunk_sizes, dim, streams)) RuntimeError: CUDA error: invalid device ordinal (exchangeDevice at /pytorch/c10/cuda/impl/CUDAGuardImpl.h:29) frame #0: std::function<std::string ()>::operator()() const + 0x11 (0x7fc16c36c441 in /home/teamway/.local/lib/python3.6/site-packages/torch/lib/libc10.so) frame #1: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x2a (0x7fc16c36bd7a in /home/teamway/.local/lib/python3.6/site-packages/torch/lib/libc10.so) frame #2: + 0xb814 (0x7fc169e51814 in /home/teamway/.local/lib/python3.6/site-packages/torch/lib/libc10_cuda.so) frame #3: at::CUDAType::empty(c10::ArrayRef, c10::TensorOptions const&) const + 0x135 (0x7fc177e6a695 in /home/teamway/.local/lib/python3.6/site-packages/torch/lib/libcaffe2_gpu.so) frame #4: torch::autograd::VariableType::empty(c10::ArrayRef, c10::TensorOptions const&) const + 0x284 (0x7fc16ad25094 in /home/teamway/.local/lib/python3.6/site-packages/torch/lib/libtorch.so.1) frame #5: at::native::to(at::Tensor const&, c10::TensorOptions const&, bool, bool) + 0x506 (0x7fc16cf91666 in /home/teamway/.local/lib/python3.6/site-packages/torch/lib/libcaffe2.so) frame #6: at::TypeDefault::to(at::Tensor const&, c10::TensorOptions const&, bool, bool) const + 0x17 (0x7fc16d210857 in /home/teamway/.local/lib/python3.6/site-packages/torch/lib/libcaffe2.so) frame #7: torch::autograd::VariableType::to(at::Tensor const&, c10::TensorOptions const&, bool, bool) const + 0x2c2 (0x7fc16ac0eb52 in /home/teamway/.local/lib/python3.6/site-packages/torch/lib/libtorch.so.1) frame #8: torch::cuda::scatter(at::Tensor const&, c10::ArrayRef, c10::optional<std::vector<long, std::allocator > > const&, long, c10::optional<std::vector<c10::optional, std::allocator<c10::optional > > > const&) + 0x389 (0x7fc16b162fd9 in /home/teamway/.local/lib/python3.6/site-packages/torch/lib/libtorch.so.1) frame #9: + 0x5a27bf (0x7fc1abadd7bf in /home/teamway/.local/lib/python3.6/site-packages/torch/lib/libtorch_python.so) frame #10: + 0x130cfc (0x7fc1ab66bcfc in /home/teamway/.local/lib/python3.6/site-packages/torch/lib/libtorch_python.so) frame #11: /usr/bin/python() [0x5030d5] frame #12: _PyEval_EvalFrameDefault + 0x449 (0x506859 in /usr/bin/python) frame #13: /usr/bin/python() [0x504c28] frame #14: /usr/bin/python() [0x502540] frame #15: /usr/bin/python() [0x502f3d] frame #16: _PyEval_EvalFrameDefault + 0x449 (0x506859 in /usr/bin/python) frame #17: /usr/bin/python() [0x504c28] frame #18: /usr/bin/python() [0x58644b] frame #19: PyObject_Call + 0x3e (0x59ebbe in /usr/bin/python) frame #20: THPFunction_apply(_object, _object*) + 0x6b1 (0x7fc1ab8ee481 in /home/teamway/.local/lib/python3.6/site-packages/torch/lib/libtorch_python.so) frame #21: /usr/bin/python() [0x502d6f] frame #22: _PyEval_EvalFrameDefault + 0x449 (0x506859 in /usr/bin/python) frame #23: /usr/bin/python() [0x504c28] frame #24: _PyFunction_FastCallDict + 0x2de (0x501b2e in /usr/bin/python) frame #25: _PyObject_FastCallDict + 0x4f1 (0x5a36f1 in /usr/bin/python) frame #26: /usr/bin/python() [0x50f011] frame #27: PySequence_Tuple + 0x182 (0x5a0502 in /usr/bin/python) frame #28: _PyEval_EvalFrameDefault + 0x5869 (0x50bc79 in /usr/bin/python) frame #29: /usr/bin/python() [0x504c28] frame #30: _PyFunction_FastCallDict + 0x2de (0x501b2e in /usr/bin/python) frame #31: _PyObject_FastCallDict + 0x4f1 (0x5a36f1 in /usr/bin/python) frame #32: /usr/bin/python() [0x50f011] frame #33: PySequence_Tuple + 0x182 (0x5a0502 in /usr/bin/python) frame #34: _PyEval_EvalFrameDefault + 0x5869 (0x50bc79 in /usr/bin/python) frame #35: /usr/bin/python() [0x504c28] frame #36: /usr/bin/python() [0x502540] frame #37: /usr/bin/python() [0x502f3d] frame #38: _PyEval_EvalFrameDefault + 0x449 (0x506859 in /usr/bin/python) frame #39: /usr/bin/python() [0x504c28] frame #40: /usr/bin/python() [0x502540] frame #41: /usr/bin/python() [0x502f3d] frame #42: _PyEval_EvalFrameDefault + 0x449 (0x506859 in /usr/bin/python) frame #43: /usr/bin/python() [0x504c28] frame #44: /usr/bin/python() [0x502540] frame #45: /usr/bin/python() [0x502f3d] frame #46: _PyEval_EvalFrameDefault + 0x1231 (0x507641 in /usr/bin/python) frame #47: /usr/bin/python() [0x502209] frame #48: /usr/bin/python() [0x502f3d] frame #49: _PyEval_EvalFrameDefault + 0x449 (0x506859 in /usr/bin/python) frame #50: /usr/bin/python() [0x504c28] frame #51: _PyFunction_FastCallDict + 0x2de (0x501b2e in /usr/bin/python) frame #52: /usr/bin/python() [0x591461] frame #53: PyObject_Call + 0x3e (0x59ebbe in /usr/bin/python) frame #54: _PyEval_EvalFrameDefault + 0x1807 (0x507c17 in /usr/bin/python) frame #55: /usr/bin/python() [0x504c28] frame #56: _PyFunction_FastCallDict + 0x2de (0x501b2e in /usr/bin/python) frame #57: /usr/bin/python() [0x591461] frame #58: PyObject_Call + 0x3e (0x59ebbe in /usr/bin/python) frame #59: /usr/bin/python() [0x54d4e2] frame #60: _PyObject_FastCallKeywords + 0x19c (0x5a730c in /usr/bin/python) frame #61: /usr/bin/python() [0x503073] frame #62: _PyEval_EvalFrameDefault + 0x449 (0x506859 in /usr/bin/python) frame #63: /usr/bin/python() [0x504c28]

Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/media/data/liuben/CornerNet-Lite/train.py", line 69, in pin_memory data = data_queue.get() File "/usr/lib/python3.6/multiprocessing/queues.py", line 113, in get return _ForkingPickler.loads(res) File "/home/teamway/.local/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 276, in rebuild_storage_fd fd = df.detach() File "/usr/lib/python3.6/multiprocessing/resource_sharer.py", line 57, in detach with _resource_sharer.get_connection(self._id) as conn: File "/usr/lib/python3.6/multiprocessing/resource_sharer.py", line 87, in get_connection c = Client(address, authkey=process.current_process().authkey) File "/usr/lib/python3.6/multiprocessing/connection.py", line 493, in Client answer_challenge(c, authkey) File "/usr/lib/python3.6/multiprocessing/connection.py", line 732, in answer_challenge message = connection.recv_bytes(256) # reject large message File "/usr/lib/python3.6/multiprocessing/connection.py", line 216, in recv_bytes buf = self._recv_bytes(maxlength) File "/usr/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes buf = self._recv(4) File "/usr/lib/python3.6/multiprocessing/connection.py", line 379, in _recv chunk = read(handle, remaining) ConnectionResetError: [Errno 104] Connection reset by peer

Process finished with exit code 1

[chaoshengzhe]

/usr/bin/python /media/data/liuben/CornerNet-Lite/train.py Process 0: loading all datasets... Process 0: using 4 workers loading annotations into memory... Done (t=0.00s) creating index... index created! loading annotations into memory... Done (t=0.00s) creating index... index created! loading annotations into memory... Done (t=0.00s) creating index... index created! loading annotations into memory... Done (t=0.00s) creating index... index created! loading annotations into memory... Done (t=0.00s) creating index... index created! system config... {'batch_size': 48, 'cache_dir': './cache', 'chunk_sizes': [12, 12, 12, 12], 'config_dir': './config', 'data_dir': './data', 'data_rng': <mtrand.RandomState object at 0x7fc160b361f8>, 'dataset': 'COCO', 'decay_rate': 10, 'display': 5, 'learning_rate': 0.00025, 'max_iter': 500000, 'nnet_rng': <mtrand.RandomState object at 0x7fc160b36240>, 'opt_algo': 'adam', 'prefetch_size': 5, 'pretrain': None, 'result_dir': './results', 'sampling_function': 'cornernet_saccade', 'snapshot': 5000, 'snapshot_name': 'CornerNet_Saccade', 'stepsize': 450000, 'test_split': 'testdev', 'train_split': 'trainval', 'val_iter': 100, 'val_split': 'minival'} db config... {'ae_threshold': 0.3, 'att_max_crops': 30, 'att_nms_ks': [3, 3, 3], 'att_ranges': [[96, 256], [32, 96], [0, 32]], 'att_ratios': [16, 8, 4], 'att_scales': [[1, 2, 4]], 'att_sizes': [[16, 16], [32, 32], [64, 64]], 'att_thresholds': [0.3], 'border': 64, 'categories': 2, 'data_aug': True, 'gaussian_bump': True, 'gaussian_iou': 0.5, 'gaussian_radius': -1, 'init_sizes': [192, 255], 'input_size': [255, 255], 'lighting': True, 'max_per_image': 100, 'max_per_set': 40, 'max_scale': 32, 'merge_bbox': False, 'min_scale': 16, 'nms_algorithm': 'exp_soft_nms', 'nms_kernel': 3, 'nms_threshold': 0.5, 'num_dets': 12, 'output_sizes': [[64, 64]], 'rand_center': True, 'rand_color': False, 'rand_crop': False, 'rand_scale_max': 1.1, 'rand_scale_min': 0.5, 'rand_scale_step': 0.1, 'rand_scales': array([0.5, 0.6, 0.7, 0.8, 0.9, 1. , 1.1]), 'ref_dets': True, 'score_threshold': 0.05, 'test_flipped': True, 'test_scales': [1], 'top_k': 12, 'view_sizes': [], 'weight_exp': 8} len of db: 25 distributed: False Process 0: building model... total parameters: 116969339 start prefetching data... shuffling indices... start prefetching data... shuffling indices... start prefetching data... shuffling indices... start prefetching data... shuffling indices... start prefetching data... setting learning rate to: 0.00025 training start... shuffling indices... 0%| | 0/500000 [00:00<?, ?it/s]shuffling indices... shuffling indices... shuffling indices... shuffling indices... shuffling indices... shuffling indices... shuffling indices... shuffling indices... shuffling indices... shuffling indices... shuffling indices... shuffling indices... shuffling indices... shuffling indices... shuffling indices... shuffling indices... shuffling indices...

shuffling indices... Traceback (most recent call last): File "/media/data/liuben/CornerNet-Lite/train.py", line 250, in main(None, ngpus_per_node, args) File "/media/data/liuben/CornerNet-Lite/train.py", line 234, in main train(training_dbs, validation_db, system_config, model, args) File "/media/data/liuben/CornerNet-Lite/train.py", line 166, in train training_loss = nnet.train(training) File "/media/data/liuben/CornerNet-Lite/core/nnet/py_factory.py", line 93, in train loss = self.network(xs, ys) File "/home/teamway/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call* result = self.forward(input, **kwargs) File "/media/data/liuben/CornerNet-Lite/core/models/py_utils/data_parallel.py", line 66, in forward inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids, self.chunk_sizes) File "/media/data/liuben/CornerNet-Lite/core/models/py_utils/data_parallel.py", line 77, in scatter return scatter_kwargs(inputs, kwargs, device_ids, dim=self.dim, chunk_sizes=self.chunk_sizes) File "/media/data/liuben/CornerNet-Lite/core/models/py_utils/scatter_gather.py", line 30, in scatter_kwargs inputs = scatter(inputs, target_gpus, dim, chunk_sizes) if inputs else [] File "/media/data/liuben/CornerNet-Lite/core/models/py_utils/scatter_gather.py", line 25, in scatter return scatter_map(inputs) File "/media/data/liuben/CornerNet-Lite/core/models/py_utils/scatter_gather.py", line 18, in scatter_map return list(zip(_map(scatter_map, obj))) File "/media/data/liuben/CornerNet-Lite/core/models/py_utils/scatter_gather.py", line 20, in scatter_map return list(map(list, zip(_map(scatter_map, obj)))) File "/media/data/liuben/CornerNet-Lite/core/models/py_utils/scatter_gather.py", line 15, in scatter_map return Scatter.apply(target_gpus, chunk_sizes, dim, obj) File "/home/teamway/.local/lib/python3.6/site-packages/torch/nn/parallel/_functions.py", line 89, in forward outputs = comm.scatter(input, target_gpus, chunk_sizes, ctx.dim, streams) File "/home/teamway/.local/lib/python3.6/site-packages/torch/cuda/comm.py", line 147, in scatter return tuple(torch._C._scatter(tensor, devices, chunk_sizes, dim, streams)) RuntimeError: CUDA error: invalid device ordinal (exchangeDevice at /pytorch/c10/cuda/impl/CUDAGuardImpl.h:29) frame #0: std::function<std::string ()>::operator()() const + 0x11 (0x7fc16c36c441 in /home/teamway/.local/lib/python3.6/site-packages/torch/lib/libc10.so) frame #1: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x2a (0x7fc16c36bd7a in /home/teamway/.local/lib/python3.6/site-packages/torch/lib/libc10.so) frame #2: + 0xb814 (0x7fc169e51814 in /home/teamway/.local/lib/python3.6/site-packages/torch/lib/libc10_cuda.so) frame #3: at::CUDAType::empty(c10::ArrayRef, c10::TensorOptions const&) const + 0x135 (0x7fc177e6a695 in /home/teamway/.local/lib/python3.6/site-packages/torch/lib/libcaffe2_gpu.so) frame #4: torch::autograd::VariableType::empty(c10::ArrayRef, c10::TensorOptions const&) const + 0x284 (0x7fc16ad25094 in /home/teamway/.local/lib/python3.6/site-packages/torch/lib/libtorch.so.1) frame #5: at::native::to(at::Tensor const&, c10::TensorOptions const&, bool, bool) + 0x506 (0x7fc16cf91666 in /home/teamway/.local/lib/python3.6/site-packages/torch/lib/libcaffe2.so) frame #6: at::TypeDefault::to(at::Tensor const&, c10::TensorOptions const&, bool, bool) const + 0x17 (0x7fc16d210857 in /home/teamway/.local/lib/python3.6/site-packages/torch/lib/libcaffe2.so) frame #7: torch::autograd::VariableType::to(at::Tensor const&, c10::TensorOptions const&, bool, bool) const + 0x2c2 (0x7fc16ac0eb52 in /home/teamway/.local/lib/python3.6/site-packages/torch/lib/libtorch.so.1) frame #8: torch::cuda::scatter(at::Tensor const&, c10::ArrayRef, c10::optional<std::vector<long, std::allocator > > const&, long, c10::optional<std::vector<c10::optionalc10::cuda::CUDAStream, std::allocator > > const&) + 0x389 (0x7fc16b162fd9 in /home/teamway/.local/lib/python3.6/site-packages/torch/lib/libtorch.so.1) frame #9: + 0x5a27bf (0x7fc1abadd7bf in /home/teamway/.local/lib/python3.6/site-packages/torch/lib/libtorch_python.so) frame #10: + 0x130cfc (0x7fc1ab66bcfc in /home/teamway/.local/lib/python3.6/site-packages/torch/lib/libtorch_python.so) frame #11: /usr/bin/python() [0x5030d5] frame #12: _PyEval_EvalFrameDefault + 0x449 (0x506859 in /usr/bin/python) frame #13: /usr/bin/python() [0x504c28] frame #14: /usr/bin/python() [0x502540] frame #15: /usr/bin/python() [0x502f3d] frame #16: _PyEval_EvalFrameDefault + 0x449 (0x506859 in /usr/bin/python) frame #17: /usr/bin/python() [0x504c28] frame #18: /usr/bin/python() [0x58644b] frame #19: PyObject_Call + 0x3e (0x59ebbe in /usr/bin/python) frame #20: THPFunction_apply(object, object) + 0x6b1 (0x7fc1ab8ee481 in /home/teamway/.local/lib/python3.6/site-packages/torch/lib/libtorch_python.so) frame #21: /usr/bin/python() [0x502d6f] frame #22: _PyEval_EvalFrameDefault + 0x449 (0x506859 in /usr/bin/python) frame #23: /usr/bin/python() [0x504c28] frame #24: _PyFunction_FastCallDict + 0x2de (0x501b2e in /usr/bin/python) frame #25: _PyObject_FastCallDict + 0x4f1 (0x5a36f1 in /usr/bin/python) frame #26: /usr/bin/python() [0x50f011] frame #27: PySequence_Tuple + 0x182 (0x5a0502 in /usr/bin/python) frame #28: _PyEval_EvalFrameDefault + 0x5869 (0x50bc79 in /usr/bin/python) frame #29: /usr/bin/python() [0x504c28] frame #30: _PyFunction_FastCallDict + 0x2de (0x501b2e in /usr/bin/python) frame #31: _PyObject_FastCallDict + 0x4f1 (0x5a36f1 in /usr/bin/python) frame #32: /usr/bin/python() [0x50f011] frame #33: PySequence_Tuple + 0x182 (0x5a0502 in /usr/bin/python) frame #34: _PyEval_EvalFrameDefault + 0x5869 (0x50bc79 in /usr/bin/python) frame #35: /usr/bin/python() [0x504c28] frame #36: /usr/bin/python() [0x502540] frame #37: /usr/bin/python() [0x502f3d] frame #38: _PyEval_EvalFrameDefault + 0x449 (0x506859 in /usr/bin/python) frame #39: /usr/bin/python() [0x504c28] frame #40: /usr/bin/python() [0x502540] frame #41: /usr/bin/python() [0x502f3d] frame #42: _PyEval_EvalFrameDefault + 0x449 (0x506859 in /usr/bin/python) frame #43: /usr/bin/python() [0x504c28] frame #44: /usr/bin/python() [0x502540] frame #45: /usr/bin/python() [0x502f3d] frame #46: _PyEval_EvalFrameDefault + 0x1231 (0x507641 in /usr/bin/python) frame #47: /usr/bin/python() [0x502209] frame #48: /usr/bin/python() [0x502f3d] frame #49: _PyEval_EvalFrameDefault + 0x449 (0x506859 in /usr/bin/python) frame #50: /usr/bin/python() [0x504c28] frame #51: _PyFunction_FastCallDict + 0x2de (0x501b2e in /usr/bin/python) frame #52: /usr/bin/python() [0x591461] frame #53: PyObject_Call + 0x3e (0x59ebbe in /usr/bin/python) frame #54: _PyEval_EvalFrameDefault + 0x1807 (0x507c17 in /usr/bin/python) frame #55: /usr/bin/python() [0x504c28] frame #56: _PyFunction_FastCallDict + 0x2de (0x501b2e in /usr/bin/python) frame #57: /usr/bin/python() [0x591461] frame #58: PyObject_Call + 0x3e (0x59ebbe in /usr/bin/python) frame #59: /usr/bin/python() [0x54d4e2] frame #60: _PyObject_FastCallKeywords + 0x19c (0x5a730c in /usr/bin/python) frame #61: /usr/bin/python() [0x503073] frame #62: _PyEval_EvalFrameDefault + 0x449 (0x506859 in /usr/bin/python) frame #63: /usr/bin/python() [0x504c28]

Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/media/data/liuben/CornerNet-Lite/train.py", line 69, in pin_memory data = data_queue.get() File "/usr/lib/python3.6/multiprocessing/queues.py", line 113, in get return _ForkingPickler.loads(res) File "/home/teamway/.local/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 276, in rebuild_storage_fd fd = df.detach() File "/usr/lib/python3.6/multiprocessing/resource_sharer.py", line 57, in detach with _resource_sharer.get_connection(self._id) as conn: File "/usr/lib/python3.6/multiprocessing/resource_sharer.py", line 87, in get_connection c = Client(address, authkey=process.current_process().authkey) File "/usr/lib/python3.6/multiprocessing/connection.py", line 493, in Client answer_challenge(c, authkey) File "/usr/lib/python3.6/multiprocessing/connection.py", line 732, in answer_challenge message = connection.recv_bytes(256) # reject large message File "/usr/lib/python3.6/multiprocessing/connection.py", line 216, in recv_bytes buf = self._recv_bytes(maxlength) File "/usr/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes buf = self._recv(4) File "/usr/lib/python3.6/multiprocessing/connection.py", line 379, in _recv chunk = read(handle, remaining) ConnectionResetError: [Errno 104] Connection reset by peer

Process finished with exit code 1

How can I sove the problem?

[Looson]

Have you solve this problem?

[lyj96]

Have you solve this problem?

[berieo]

Just run again