ValueError: signal number 32 out of range and the training progress bar not moving when running training command "python rotate_train.py"

hopesala commented 3 years ago

python rotate_train.py --weights rotate-yolov5s-ucas.pt --cfg rotate_yolov5s_ucas.yaml --data rotate_ucas.yaml --hyp hyp.ucas.yaml --img-size 1024 --epochs 3 --batch-size 4 --noautoanchor --rotate --cache YOLOv5 🚀 2021-8-18 torch 1.7.0 CUDA:0 (NVIDIA GeForce GTX 1080 Ti, 11170.0625MB)

Namespace(adam=False, artifact_alias='latest', batch_size=4, bbox_interval=-1, bucket='', cache_images=True, cfg='./models/rotate_yolov5s_ucas.yaml', data='./data/rotate_ucas.yaml', device='', entity=None, epochs=3, evolve=False, exist_ok=False, global_rank=-1, hyp='./data/hyp.ucas.yaml', image_weights=False, img_size=[1024, 1024], label_smoothing=0.0, linear_lr=False, local_rank=-1, multi_scale=False, name='exp', noautoanchor=True, nosave=False, notest=False, project='runs/train', quad=False, rect=False, resume=False, rotate=True, save_dir='runs/train/exp6', save_period=-1, single_cls=False, sync_bn=False, total_batch_size=4, upload_dataset=False, weights='rotate-yolov5s-ucas.pt', workers=4, world_size=1) tensorboard: Start with 'tensorboard --logdir runs/train', view at http://localhost:6006/ hyperparameters: lr0=0.01, lrf=0.2, momentum=0.937, weight_decay=0.0005, warmup_epochs=3.0, warmup_momentum=0.8, warmup_bias_lr=0.1, box=0.05, cls=0.5, cls_pw=1.0, obj=1.0, obj_pw=1.0, iou_t=0.1, anchor_t=4.0, fl_gamma=0.0, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, degrees=30.0, translate=0.1, scale=0.0, shear=5.0, perspective=0.0005, flipud=0.5, fliplr=0.5, mosaic=0.0, mixup=0.0 wandb: Install Weights & Biases for YOLOv5 logging with 'pip install wandb' (recommended)

             from  n    params  module                                  arguments

0 -1 1 1 -1 1 2 -1 1 3 -1 1 4 5 6 7 8 9 10 11 -1 1 12 [-1, 6] 1 13 14 15 -1 1 16 [-1, 4] 1 17 18 19 [-1, 14] 1 20 21 22 [-1, 10] 1 23 24 [17, 20, 23] 1 Model Summary: 3520 models.common.Focus [3, 32, 3]
18560 models.common.Conv [32, 64, 3, 2]
18816 models.common.C3 [64, 64, 1]
73984 models.common.Conv [64, 128, 3, 2]
-1 1 156928 models.common.C3 [128, 128, 3]
-1 1 295424 models.common.Conv [128, 256, 3, 2]
-1 1 625152 models.common.C3 [256, 256, 3]
-1 1 1180672 models.common.Conv [256, 512, 3, 2]
-1 1 656896 models.common.SPP [512, 512, [5, 9, 13]]
-1 1 1182720 models.common.C3 [512, 512, 1, False]
-1 1 131584 models.common.Conv [512, 256, 1, 1]
0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
0 models.common.Concat [1]
-1 1 361984 models.common.C3 [512, 256, 1, False]
-1 1 33024 models.common.Conv [256, 128, 1, 1]
0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
0 models.common.Concat [1]
-1 1 90880 models.common.C3 [256, 128, 1, False]
-1 1 147712 models.common.Conv [128, 128, 3, 2]
0 models.common.Concat [1]
-1 1 296448 models.common.C3 [256, 256, 1, False]
-1 1 590336 models.common.Conv [256, 256, 3, 2]
0 models.common.Concat [1]
-1 1 1182720 models.common.C3 [512, 512, 1, False]
40455 models.yolo.Rotate_Detect [2, [[27, 26, 20, 40, 44, 19, 34, 34, 25, 47], [55, 24, 44, 38, 31, 61, 50, 50, 63, 45], [65, 62, 88, 60, 84, 79, 113, 85, 148, 122]], [128, 256, 512]] 283 layers, 7087815 parameters, 7087815 gradients

Transferred 360/362 items from rotate-yolov5s-ucas.pt Scaled weight_decay = 0.0005 Optimizer groups: 62 .bias, 62 conv.weight, 59 other train: Scanning '../UCAS50/train.cache' images and labels... 38 found, 0 missing train: Caching images (0.1GB): 100%|███████████| 38/38 [00:00<00:00, 177.94it/s] val: Scanning '../UCAS50/val.cache' images and labels... 10 found, 0 missing, 0 0%| | 0/10 [00:00<?, ?it/s]Exception in thread Thread-14: Traceback (most recent call last): File "/home/cao/anaconda3/envs/stark/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/home/cao/anaconda3/envs/stark/lib/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/home/cao/anaconda3/envs/stark/lib/python3.6/multiprocessing/resource_sharer.py", line 139, in _serve signal.pthread_sigmask(signal.SIG_BLOCK, range(1, signal.NSIG)) File "/home/cao/anaconda3/envs/stark/lib/python3.6/signal.py", line 60, in pthread_sigmask sigs_set = _signal.pthread_sigmask(how, mask) ValueError: signal number 32 out of range

val: Caching images (0.0GB): 100%|██████████████| 10/10 [00:00<00:00, 99.27it/s] Exception in thread Thread-14: Traceback (most recent call last): File "/home/cao/anaconda3/envs/stark/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/home/cao/anaconda3/envs/stark/lib/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/home/cao/anaconda3/envs/stark/lib/python3.6/multiprocessing/resource_sharer.py", line 139, in _serve signal.pthread_sigmask(signal.SIG_BLOCK, range(1, signal.NSIG)) File "/home/cao/anaconda3/envs/stark/lib/python3.6/signal.py", line 60, in pthread_sigmask sigs_set = _signal.pthread_sigmask(how, mask) ValueError: signal number 32 out of range

Exception in thread Thread-14: Traceback (most recent call last): File "/home/cao/anaconda3/envs/stark/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/home/cao/anaconda3/envs/stark/lib/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/home/cao/anaconda3/envs/stark/lib/python3.6/multiprocessing/resource_sharer.py", line 139, in _serve signal.pthread_sigmask(signal.SIG_BLOCK, range(1, signal.NSIG)) File "/home/cao/anaconda3/envs/stark/lib/python3.6/signal.py", line 60, in pthread_sigmask sigs_set = _signal.pthread_sigmask(how, mask) ValueError: signal number 32 out of range

Plotting labels... Exception in thread Thread-26: Traceback (most recent call last): File "/home/cao/anaconda3/envs/stark/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/home/cao/anaconda3/envs/stark/lib/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/home/cao/anaconda3/envs/stark/lib/python3.6/multiprocessing/resource_sharer.py", line 139, in _serve signal.pthread_sigmask(signal.SIG_BLOCK, range(1, signal.NSIG)) File "/home/cao/anaconda3/envs/stark/lib/python3.6/signal.py", line 60, in pthread_sigmask sigs_set = _signal.pthread_sigmask(how, mask) ValueError: signal number 32 out of range

Exception in thread Thread-26: Traceback (most recent call last): File "/home/cao/anaconda3/envs/stark/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/home/cao/anaconda3/envs/stark/lib/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/home/cao/anaconda3/envs/stark/lib/python3.6/multiprocessing/resource_sharer.py", line 139, in _serve signal.pthread_sigmask(signal.SIG_BLOCK, range(1, signal.NSIG)) File "/home/cao/anaconda3/envs/stark/lib/python3.6/signal.py", line 60, in pthread_sigmask sigs_set = _signal.pthread_sigmask(how, mask) ValueError: signal number 32 out of range

Image sizes 1024 train, 1024 test Using 4 dataloader workers Logging results to runs/train/exp6 Starting training for 3 epochs...

 Epoch   gpu_mem       box       obj       cls     total    labels  img_size

0%| | 0/10 [00:00<?, ?it/s]

What could be the reason and how to fix this problem? thanks for replying~

hopesala commented 3 years ago

Besides, when I run command "python rotate_detect.py --weights rotate-yolov5s-ucas.pt --img 1024 --conf 0.75 \ --source data/images/UCAS-AOD --iou-thres 0.4 --hide-labels" no objects are detected(no red rectangles are found in the runs/detect/expX folder output imgs)

XinzeLee commented 3 years ago

It seems like a bug in python. Please refer to this link for detailed explainations.

XinzeLee / RotateObjectDetection

ValueError: signal number 32 out of range and the training progress bar not moving when running training command "python rotate_train.py" #10