uci-cbcl / NoduleNet

[MICCAI' 19] NoduleNet: Decoupled False Positive Reduction for Pulmonary Nodule Detection and Segmentation
Other
187 stars 58 forks source link

Training time is too long, and the GPU utilization is very low ? #33

Closed sUtomorrow closed 3 years ago

sUtomorrow commented 3 years ago

_

scripts $ ./cross_val_6fold.sh Warning: C++ module import failed! This should only happen in deployment Warning: C++ module import failed! This should only happen in deployment Warning: C++ module import failed! This should only happen in deployment Warning: C++ module import failed! This should only happen in deployment [Training configuration] net NoduleNet epochs 200 batch_size 16 epoch_rcnn 65 epoch_mask 80 ckpt None optimizer SGD init_lr 0.01 momentum 0.9 weight_decay 0.0001 epoch_save 1 out_dir results/cross_val_test/0_mask train_set_list ['split/cross_val/0_train.csv'] val_set_list ['split/cross_val/0_val.csv'] data_dir /opt/tiger/data/LUNA16/preprocessed_test/3 num_workers 16 [Model configuration] {'anchors': [[5, 5, 5], [10, 10, 10], [20, 20, 20], [30, 30, 30], [50, 50, 50]], 'augtype': {'flip': True, 'rotate': True, 'scale': True, 'swap': False}, 'blacklist': [], 'bound_size': 12, 'box_reg_weight': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0], 'chanel': 1, 'crop_size': [128, 128, 128], 'mask_crop_size': [24, 48, 48], 'mask_test_nms_overlap_threshold': 0.3, 'max_stride': 16, 'num_class': 2, 'num_hard': 3, 'num_neg': 800, 'pad_value': 170, 'r_rand_crop': 0.0, 'rcnn_crop_size': (7, 7, 7), 'rcnn_test_nms_overlap_threshold': 0.1, 'rcnn_test_nms_pre_score_threshold': 0.0, 'rcnn_train_batch_size': 64, 'rcnn_train_bg_thresh_high': 0.1, 'rcnn_train_fg_fraction': 0.5, 'rcnn_train_fg_thresh_low': 0.5, 'rcnn_train_nms_overlap_threshold': 0.1, 'rcnn_train_nms_pre_score_threshold': 0.5, 'rpn_test_nms_overlap_threshold': 0.1, 'rpn_test_nms_pre_score_threshold': 0.5, 'rpn_train_bg_thresh_high': 0.02, 'rpn_train_fg_thresh_low': 0.5, 'rpn_train_nms_num': 300, 'rpn_train_nms_overlap_threshold': 0.1, 'rpn_train_nms_pre_score_threshold': 0.5, 'stride': 4, 'th_neg': 0.02, 'th_pos_train': 0.5, 'th_pos_val': 1} [start_epoch 1, out_dir results/cross_val_test/0_mask] [length of train loader 59, length of valid loader 12] Total: 0%| | 0/200 [00:00<?, ?it/s][epoch 1, lr 0.010000, use_rcnn: False, use_mask: False] train.py:2 16: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray truth_box = np.array(truth_box) train.py:217: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray truth_label = np.array(truth_label) train.py:218: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray truth_mask = np.array(truth_mask)

Train 1: 31%|███████████████████████████████▍ | 18/59 [31:46<1:10:45, 103.55s/it] Train 1: 44%|██████████████████████████████████████████████▎ | 26/59 [45:16<56:07, 102.05s/it]

_