H-Justus
commented
1 year ago Could you provide some details?
Command Line Args: Namespace(config_file='projects/SWINTS/configs/SWINTS-swin-mixtrain.yaml', dist_url='tcp://127.0.0.1:50154', eval_only=False, machine_rank=0, num_gpus=2, num_machines=1, opts=[], resume=False)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, kwargs)
/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, **kwargs)
/opt/conda/conda-bld/pytorch_1616554788289/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: ClassNLLCriterion_updateOutput_no_reduce_kernel: block: [0,0,0], thread: [0,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed.
/opt/conda/conda-bld/pytorch_1616554788289/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: ClassNLLCriterion_updateOutput_no_reduce_kernel: block: [0,0,0], thread: [1,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed.
/opt/conda/conda-bld/pytorch_1616554788289/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: ClassNLLCriterion_updateOutput_no_reduce_kernel: block: [0,0,0], thread: [2,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed.
/opt/conda/conda-bld/pytorch_1616554788289/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: ClassNLLCriterion_updateOutput_no_reduce_kernel: block: [0,0,0], thread: [3,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed.
/opt/conda/conda-bld/pytorch_1616554788289/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: ClassNLLCriterion_updateOutput_no_reduce_kernel: block: [0,0,0], thread: [4,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed.
/opt/conda/conda-bld/pytorch_1616554788289/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: ClassNLLCriterion_updateOutput_no_reduce_kernel: block: [0,0,0], thread: [5,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed.
/opt/conda/conda-bld/pytorch_1616554788289/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: ClassNLLCriterion_updateOutput_no_reduce_kernel: block: [0,0,0], thread: [6,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed.
/opt/conda/conda-bld/pytorch_1616554788289/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: ClassNLLCriterion_updateOutput_no_reduce_kernel: block: [0,0,0], thread: [7,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed.
/opt/conda/conda-bld/pytorch_1616554788289/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: ClassNLLCriterion_updateOutput_no_reduce_kernel: block: [0,0,0], thread: [0,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed.
/opt/conda/conda-bld/pytorch_1616554788289/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: ClassNLLCriterion_updateOutput_no_reduce_kernel: block: [0,0,0], thread: [1,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed.
/opt/conda/conda-bld/pytorch_1616554788289/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: ClassNLLCriterion_updateOutput_no_reduce_kernel: block: [0,0,0], thread: [2,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed.
/opt/conda/conda-bld/pytorch_1616554788289/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: ClassNLLCriterion_updateOutput_no_reduce_kernel: block: [0,0,0], thread: [3,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed.
/opt/conda/conda-bld/pytorch_1616554788289/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: ClassNLLCriterion_updateOutput_no_reduce_kernel: block: [0,0,0], thread: [4,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed.
/opt/conda/conda-bld/pytorch_1616554788289/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: ClassNLLCriterion_updateOutput_no_reduce_kernel: block: [0,0,0], thread: [5,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed.
/opt/conda/conda-bld/pytorch_1616554788289/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: ClassNLLCriterion_updateOutput_no_reduce_kernel: block: [0,0,0], thread: [6,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed.
/opt/conda/conda-bld/pytorch_1616554788289/work/aten/src/THCUNN/ClassNLLCriterion.cu:59: ClassNLLCriterion_updateOutput_no_reduce_kernel: block: [0,0,0], thread: [7,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed.
Traceback (most recent call last):
File "projects/SWINTS/train_net.py", line 135, in
-- Process 0 terminated with the following error: Traceback (most recent call last): File "/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap fn(i, args) File "/data/huangzixuan/SwinTextSpotter/detectron2/engine/launch.py", line 125, in _distributed_worker main_func(args) File "/data/huangzixuan/SwinTextSpotter/projects/SWINTS/train_net.py", line 129, in main return trainer.train() File "/data/huangzixuan/SwinTextSpotter/detectron2/engine/defaults.py", line 431, in train super().train(self.start_iter, self.max_iter) File "/data/huangzixuan/SwinTextSpotter/detectron2/engine/train_loop.py", line 140, in train self.run_step() File "/data/huangzixuan/SwinTextSpotter/detectron2/engine/defaults.py", line 441, in run_step self._trainer.run_step() File "/data/huangzixuan/SwinTextSpotter/detectron2/engine/train_loop.py", line 234, in run_step loss_dict = self.model(data) File "/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, kwargs) File "/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 705, in forward output = self.module(*inputs[0], *kwargs[0]) File "/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(input, kwargs) File "/data/huangzixuan/SwinTextSpotter/projects/SWINTS/swints/swints.py", line 178, in forward outputs_class, outputs_coord, outputs_mask,out_rec = self.head(features, proposal_boxes, proposal_feats, targets, mask_encoding=self.mask_encoding, matcher=self.matcher) File "/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, kwargs) File "/data/huangzixuan/SwinTextSpotter/projects/SWINTS/swints/head.py", line 258, in forward rec_result = self.rec_stage(rec_map, rec_proposal_features, gt_masks, N, nr_boxes, idx, target_rec) File "/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, *kwargs) File "/data/huangzixuan/SwinTextSpotter/projects/SWINTS/swints/rec_stage.py", line 184, in forward attn_vecs = self.seq_decoder(k_rec, targets, targets) File "/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(input, kwargs) File "/data/huangzixuan/SwinTextSpotter/projects/SWINTS/swints/roi_seq_predictors.py", line 120, in forward decoder_output, decoder_hidden, decoder_attention = self.seq_decoder( File "/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, *kwargs) File "/data/huangzixuan/SwinTextSpotter/projects/SWINTS/swints/roi_seq_predictors.py", line 375, in forward output = F.log_softmax(self.out(hidden), dim=1) File "/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(input, **kwargs) File "/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 94, in forward return F.linear(input, self.weight, self.bias) File "/home/huangzixuan/anaconda3/envs/sts/lib/python3.8/site-packages/torch/nn/functional.py", line 1753, in linear return torch._C._nn.linear(input, weight, bias) RuntimeError: CUDA error: device-side assert triggered
Could you provide some details?