search

facebookresearch / Detectron

FAIR's research platform for object detection research, implementing popular algorithms like Mask R-CNN and RetinaNet.
Apache License 2.0
26.28k stars 5.46k forks source link

VGG16 with FPN issue: Error details mentioned. Should I need to add other concat layers respectively? Where actually? #730

Closed mamunir closed 5 years ago

mamunir commented 6 years ago

Found Detectron ops lib: /usr/local/lib/libcaffe2_detectron_ops_gpu.so E1031 22:55:04.712018 19990 init_intrinsics_check.cc:43] CPU feature avx is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU. E1031 22:55:04.712038 19990 init_intrinsics_check.cc:43] CPU feature avx2 is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU. E1031 22:55:04.712041 19990 init_intrinsics_check.cc:43] CPU feature fma is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU. INFO train_net.py: 95: Called with args: INFO train_net.py: 96: Namespace(cfg_file='configs/getting_started/tutorial_1gpu_e2e_faster_rcnn_VGG-16-FPN_VOC.yaml', multi_gpu_testing=False, opts=['OUTPUT_DIR', 'experiments/output'], skip_test=False) INFO train_net.py: 102: Training with config: INFO train_net.py: 103: {'BBOX_XFORM_CLIP': 4.135166556742356, 'CLUSTER': {'ON_CLUSTER': False}, 'DATA_LOADER': {'BLOBS_QUEUE_CAPACITY': 8, 'MINIBATCH_QUEUE_SIZE': 64, 'NUM_THREADS': 4}, 'DEDUP_BOXES': 0.0625, 'DOWNLOAD_CACHE': '/tmp/detectron-download-cache', 'EPS': 1e-14, 'EXPECTED_RESULTS': [], 'EXPECTED_RESULTS_ATOL': 0.005, 'EXPECTED_RESULTS_EMAIL': '', 'EXPECTED_RESULTS_RTOL': 0.1, 'FAST_RCNN': {'CONV_HEAD_DIM': 256, 'MLP_HEAD_DIM': 1024, 'NUM_STACKED_CONVS': 4, 'ROI_BOX_HEAD': 'VGG16.add_VGG16_roi_fc_head', 'ROI_XFORM_METHOD': 'RoIAlign', 'ROI_XFORM_RESOLUTION': 7, 'ROI_XFORM_SAMPLING_RATIO': 2}, 'FPN': {'COARSEST_STRIDE': 32, 'DIM': 128, 'EXTRA_CONV_LEVELS': False, 'FPN_ON': True, 'MULTILEVEL_ROIS': True, 'MULTILEVEL_RPN': True, 'ROI_CANONICAL_LEVEL': 4, 'ROI_CANONICAL_SCALE': 224, 'ROI_MAX_LEVEL': 3, 'ROI_MIN_LEVEL': 2, 'RPN_ANCHOR_START_SIZE': 32, 'RPN_ASPECT_RATIOS': (0.5, 1, 2), 'RPN_MAX_LEVEL': 3, 'RPN_MIN_LEVEL': 2, 'USE_GN': False, 'ZERO_INIT_LATERAL': False}, 'GROUP_NORM': {'DIM_PER_GP': -1, 'EPSILON': 1e-05, 'NUM_GROUPS': 32}, 'KRCNN': {'CONV_HEAD_DIM': 256, 'CONV_HEAD_KERNEL': 3, 'CONV_INIT': 'GaussianFill', 'DECONV_DIM': 256, 'DECONV_KERNEL': 4, 'DILATION': 1, 'HEATMAP_SIZE': -1, 'INFERENCE_MIN_SIZE': 0, 'KEYPOINT_CONFIDENCE': 'bbox', 'LOSS_WEIGHT': 1.0, 'MIN_KEYPOINT_COUNT_FOR_VALID_MINIBATCH': 20, 'NMS_OKS': False, 'NORMALIZE_BY_VISIBLE_KEYPOINTS': True, 'NUM_KEYPOINTS': -1, 'NUM_STACKED_CONVS': 8, 'ROI_KEYPOINTS_HEAD': '', 'ROI_XFORM_METHOD': 'RoIAlign', 'ROI_XFORM_RESOLUTION': 7, 'ROI_XFORM_SAMPLING_RATIO': 0, 'UP_SCALE': -1, 'USE_DECONV': False, 'USE_DECONV_OUTPUT': False}, 'MATLAB': 'matlab', 'MEMONGER': True, 'MEMONGER_SHARE_ACTIVATIONS': False, 'MODEL': {'BBOX_REG_WEIGHTS': (10.0, 10.0, 5.0, 5.0), 'CLS_AGNOSTIC_BBOX_REG': False, 'CONV_BODY': 'FPN.add_fpn_VGG', 'EXECUTION_TYPE': 'dag', 'FASTER_RCNN': True, 'KEYPOINTS_ON': False, 'MASK_ON': False, 'NUM_CLASSES': 3, 'RPN_ONLY': False, 'TYPE': 'generalized_rcnn'}, 'MRCNN': {'CLS_SPECIFIC_MASK': True, 'CONV_INIT': 'GaussianFill', 'DILATION': 2, 'DIM_REDUCED': 256, 'RESOLUTION': 14, 'ROI_MASK_HEAD': '', 'ROI_XFORM_METHOD': 'RoIAlign', 'ROI_XFORM_RESOLUTION': 7, 'ROI_XFORM_SAMPLING_RATIO': 0, 'THRESH_BINARIZE': 0.5, 'UPSAMPLE_RATIO': 1, 'USE_FC_OUTPUT': False, 'WEIGHT_LOSS_MASK': 1.0}, 'NUM_GPUS': 1, 'OUTPUT_DIR': 'experiments/output', 'PIXEL_MEANS': array([[[102.9801, 115.9465, 122.7717]]]), 'RESNETS': {'NUM_GROUPS': 1, 'RES5_DILATION': 1, 'SHORTCUT_FUNC': 'basic_bn_shortcut', 'STEM_FUNC': 'basic_bn_stem', 'STRIDE_1X1': True, 'TRANS_FUNC': 'bottleneck_transformation', 'WIDTH_PER_GROUP': 64}, 'RETINANET': {'ANCHOR_SCALE': 4, 'ASPECT_RATIOS': (0.5, 1.0, 2.0), 'BBOX_REG_BETA': 0.11, 'BBOX_REG_WEIGHT': 1.0, 'CLASS_SPECIFIC_BBOX': False, 'INFERENCE_TH': 0.05, 'LOSS_ALPHA': 0.25, 'LOSS_GAMMA': 2.0, 'NEGATIVE_OVERLAP': 0.4, 'NUM_CONVS': 4, 'POSITIVE_OVERLAP': 0.5, 'PRE_NMS_TOP_N': 1000, 'PRIOR_PROB': 0.01, 'RETINANET_ON': False, 'SCALES_PER_OCTAVE': 3, 'SHARE_CLS_BBOX_TOWER': False, 'SOFTMAX': False}, 'RFCN': {'PS_GRID_SIZE': 3}, 'RNG_SEED': 3, 'ROOT_DIR': '/home/akhtar/Downloads/Detectron-master/detectron', 'RPN': {'ASPECT_RATIOS': (0.5, 1, 2), 'RPN_ON': True, 'SIZES': (64, 128, 256, 512), 'STRIDE': 16}, 'SOLVER': {'BASE_LR': 0.0025, 'GAMMA': 0.1, 'LOG_LR_CHANGE_THRESHOLD': 1.1, 'LRS': [], 'LR_POLICY': 'steps_with_decay', 'MAX_ITER': 60, 'MOMENTUM': 0.9, 'SCALE_MOMENTUM': True, 'SCALE_MOMENTUM_THRESHOLD': 1.1, 'STEPS': [0, 30000, 40000], 'STEP_SIZE': 30000, 'WARM_UP_FACTOR': 0.3333333333333333, 'WARM_UP_ITERS': 500, 'WARM_UP_METHOD': u'linear', 'WEIGHT_DECAY': 0.0001, 'WEIGHT_DECAY_GN': 0.0}, 'TEST': {'BBOX_AUG': {'AREA_TH_HI': 32400, 'AREA_TH_LO': 2500, 'ASPECT_RATIOS': (), 'ASPECT_RATIO_H_FLIP': False, 'COORD_HEUR': 'UNION', 'ENABLED': False, 'H_FLIP': False, 'MAX_SIZE': 4000, 'SCALES': (), 'SCALE_H_FLIP': False, 'SCALE_SIZE_DEP': False, 'SCORE_HEUR': 'UNION'}, 'BBOX_REG': True, 'BBOX_VOTE': {'ENABLED': False, 'SCORING_METHOD': 'ID', 'SCORING_METHOD_BETA': 1.0, 'VOTE_TH': 0.8}, 'COMPETITION_MODE': True, 'DATASETS': ('voc_2007_test',), 'DETECTIONS_PER_IM': 100, 'FORCE_JSON_DATASET_EVAL': False, 'KPS_AUG': {'AREA_TH': 32400, 'ASPECT_RATIOS': (), 'ASPECT_RATIO_H_FLIP': False, 'ENABLED': False, 'HEUR': 'HM_AVG', 'H_FLIP': False, 'MAX_SIZE': 4000, 'SCALES': (), 'SCALE_H_FLIP': False, 'SCALE_SIZE_DEP': False}, 'MASK_AUG': {'AREA_TH': 32400, 'ASPECT_RATIOS': (), 'ASPECT_RATIO_H_FLIP': False, 'ENABLED': False, 'HEUR': 'SOFT_AVG', 'H_FLIP': False, 'MAX_SIZE': 4000, 'SCALES': (), 'SCALE_H_FLIP': False, 'SCALE_SIZE_DEP': False}, 'MAX_SIZE': 833, 'NMS': 0.5, 'PRECOMPUTED_PROPOSALS': False, 'PROPOSAL_FILES': (), 'PROPOSAL_LIMIT': 2000, 'RPN_MIN_SIZE': 0, 'RPN_NMS_THRESH': 0.7, 'RPN_POST_NMS_TOP_N': 1000, 'RPN_PRE_NMS_TOP_N': 1000, 'SCALE': 500, 'SCORE_THRESH': 0.05, 'SOFT_NMS': {'ENABLED': False, 'METHOD': 'linear', 'SIGMA': 0.5}, 'WEIGHTS': ''}, 'TRAIN': {'ASPECT_GROUPING': True, 'AUTO_RESUME': True, 'BATCH_SIZE_PER_IM': 256, 'BBOX_THRESH': 0.5, 'BG_THRESH_HI': 0.5, 'BG_THRESH_LO': 0.0, 'COPY_WEIGHTS': False, 'CROWD_FILTER_THRESH': 0.7, 'DATASETS': ('voc_2007_train',), 'FG_FRACTION': 0.25, 'FG_THRESH': 0.5, 'FREEZE_AT': 2, 'FREEZE_CONV_BODY': False, 'GT_MIN_AREA': -1, 'IMS_PER_BATCH': 2, 'MAX_SIZE': 833, 'PROPOSAL_FILES': (), 'RPN_BATCH_SIZE_PER_IM': 256, 'RPN_FG_FRACTION': 0.5, 'RPN_MIN_SIZE': 0, 'RPN_NEGATIVE_OVERLAP': 0.3, 'RPN_NMS_THRESH': 0.7, 'RPN_POSITIVE_OVERLAP': 0.7, 'RPN_POST_NMS_TOP_N': 2000, 'RPN_PRE_NMS_TOP_N': 2000, 'RPN_STRADDLE_THRESH': 0, 'SCALES': (500,), 'SNAPSHOT_ITERS': 20000, 'USE_FLIPPED': True, 'WEIGHTS': '/home/akhtar/Downloads/Detectron-master/detectron/VGG_ILSVRC_16_layers.pkl'}, 'USE_NCCL': False, 'VIS': False, 'VIS_TH': 0.9} experiments/output/train/voc_2007_train/generalized_rcnn INFO train.py: 139: Building model: generalized_rcnn WARNING cnn.py: 25: [====DEPRECATE WARNING====]: you are creating an object from CNNModelHelper class which will be deprecated soon. Please use ModelHelper object with brew module. For more information, please refer to caffe2.ai and python/brew.py, python/brew_test.py for more information. blob_in: [BlobReference("gpu_0/fpn_conv3_3"), BlobReference("gpu_0/fpn_conv2_2")] len(blobs_in): [BlobReference("gpu_0/fpn_conv3_3"), BlobReference("gpu_0/fpn_conv2_2")] 3 2 WARNING memonger.py: 55: NOTE: Executing memonger to optimize gradient memory I1031 22:55:04.768051 19990 memonger.cc:236] Remapping 26 using 7 shared blobs. INFO memonger.py: 97: Memonger memory optimization took 0.00202488899231 secs I1031 22:55:04.987249 19990 context_gpu.cu:321] GPU 0: 156 MB I1031 22:55:04.987267 19990 context_gpu.cu:325] Total: 156 MB I1031 22:55:05.009560 19990 context_gpu.cu:321] GPU 0: 326 MB I1031 22:55:05.009573 19990 context_gpu.cu:325] Total: 326 MB INFO train.py: 187: Loading dataset: ('voc_2007_train',) loading annotations into memory... Done (t=0.10s) creating index... index created! [2, 1] INFO roidb.py: 51: Appending horizontally-flipped training examples... INFO roidb.py: 53: Loaded dataset: voc_2007_train INFO roidb.py: 142: Filtered 0 roidb entries: 17744 -> 17744 INFO roidb.py: 73: Computing bounding-box regression targets... INFO roidb.py: 75: done INFO train.py: 191: 17744 roidb entries INFO net.py: 59: Loading weights from: /home/akhtar/Downloads/Detectron-master/detectron/VGG_ILSVRC_16_layers.pkl INFO net.py: 91: fpn_inner_conv3_3_w not found INFO net.py: 91: fpn_inner_conv3_3_b not found INFO net.py: 91: fpn_inner_conv2_2_lateral_w not found INFO net.py: 91: fpn_inner_conv2_2_lateral_b not found INFO net.py: 91: fpn_conv3_3_w not found INFO net.py: 91: fpn_conv3_3_b not found INFO net.py: 91: fpn_conv2_2_w not found INFO net.py: 91: fpn_conv2_2_b not found INFO net.py: 91: conv_rpn_fpn2_w not found INFO net.py: 91: conv_rpn_fpn2_b not found INFO net.py: 91: rpn_cls_logits_fpn2_w not found INFO net.py: 91: rpn_cls_logits_fpn2_b not found INFO net.py: 91: rpn_bbox_pred_fpn2_w not found INFO net.py: 91: rpn_bbox_pred_fpn2_b not found dst_name: gpu_0/fc6_w dst_name: gpu_0/fc6_w (4096, 6272) fc6_w (4096, 25088) dst_name: gpu_0/fc6_b dst_name: gpu_0/fc6_b (4096,) fc6_b (4096,) dst_name: gpu_0/fc7_w dst_name: gpu_0/fc7_w (4096, 4096) fc7_w (4096, 4096) dst_name: gpu_0/fc7_b dst_name: gpu_0/fc7_b (4096,) fc7_b (4096,) INFO net.py: 91: cls_score_voc_w not found INFO net.py: 91: cls_score_voc_b not found INFO net.py: 91: bbox_pred_voc_w not found INFO net.py: 91: bbox_pred_voc_b not found I1031 22:55:11.603101 19990 net_dag_utils.cc:102] Operator graph pruning prior to chain compute took: 8.1461e-05 secs INFO train.py: 175: Outputs saved to: /home/akhtar/Downloads/Detectron-master/detectron/experiments/output/train/voc_2007_train/generalized_rcnn INFO loader.py: 229: Pre-filling mini-batch queue... INFO loader.py: 234: [0/64] INFO loader.py: 234: [0/64] I1031 22:55:11.725433 20216 context_gpu.cu:321] GPU 0: 455 MB I1031 22:55:11.725473 20216 context_gpu.cu:325] Total: 455 MB INFO loader.py: 234: [0/64] I1031 22:55:11.864187 20216 context_gpu.cu:321] GPU 0: 591 MB I1031 22:55:11.864223 20216 context_gpu.cu:325] Total: 591 MB INFO loader.py: 234: [0/64] INFO loader.py: 234: [4/64] INFO loader.py: 234: [10/64] INFO loader.py: 234: [15/64] INFO loader.py: 234: [18/64] INFO loader.py: 234: [21/64] INFO loader.py: 234: [25/64] INFO loader.py: 234: [30/64] INFO loader.py: 234: [34/64] INFO loader.py: 234: [39/64] INFO loader.py: 234: [42/64] INFO loader.py: 234: [48/64] INFO loader.py: 234: [53/64] INFO loader.py: 234: [59/64] INFO loader.py: 234: [61/64] INFO detector.py: 479: Changing learning rate 0.000000 -> 0.000833 at iter 0 I1031 22:55:13.414777 19990 net_async_base.cc:435] Using specified CPU pool size: 4; NUMA node id: -1 I1031 22:55:13.414799 19990 net_async_base.cc:440] Created new CPU pool, size: 4; NUMA node id: -1 I1031 22:55:13.968763 20272 context_gpu.cu:321] GPU 0: 878 MB I1031 22:55:13.968782 20272 context_gpu.cu:325] Total: 878 MB I1031 22:55:13.969483 20272 context_gpu.cu:321] GPU 0: 1097 MB I1031 22:55:13.969492 20272 context_gpu.cu:325] Total: 1097 MB I1031 22:55:13.970178 20272 context_gpu.cu:321] GPU 0: 1259 MB I1031 22:55:13.970186 20272 context_gpu.cu:325] Total: 1259 MB I1031 22:55:14.007398 20272 context_gpu.cu:321] GPU 0: 1394 MB I1031 22:55:14.007416 20272 context_gpu.cu:325] Total: 1394 MB I1031 22:55:14.013844 20272 context_gpu.cu:321] GPU 0: 1557 MB I1031 22:55:14.013854 20272 context_gpu.cu:325] Total: 1557 MB I1031 22:55:14.033874 20272 context_gpu.cu:321] GPU 0: 1706 MB I1031 22:55:14.033885 20272 context_gpu.cu:325] Total: 1706 MB I1031 22:55:14.059420 20269 context_gpu.cu:321] GPU 0: 1934 MB I1031 22:55:14.059432 20269 context_gpu.cu:325] Total: 1934 MB I1031 22:55:14.073299 20269 context_gpu.cu:321] GPU 0: 2069 MB I1031 22:55:14.073323 20269 context_gpu.cu:325] Total: 2069 MB I1031 22:55:14.074018 20269 context_gpu.cu:321] GPU 0: 2204 MB I1031 22:55:14.074026 20269 context_gpu.cu:325] Total: 2204 MB E1031 22:55:14.095806 20270 net_async_base.cc:368] [enforce fail at spatial_narrow_as_op.cu:85] A.dim32(2) >= B.dim32(2). 108 vs 128. Input 0 height must be >= input 1 height. Error from operator: input: "gpu_0/rpn_labels_int32_wide_fpn3" input: "gpu_0/rpn_cls_logits_fpn3" output: "gpu_0/rpn_labels_int32_fpn3" name: "" type: "SpatialNarrowAs" device_option { device_type: 1 cuda_gpu_id: 0 }, op SpatialNarrowAs WARNING workspace.py: 185: Original python traceback for operator 59 in network generalized_rcnn in exception above (most recent call last): WARNING workspace.py: 190: File "tools/train_net.py", line 128, in WARNING workspace.py: 190: File "tools/train_net.py", line 110, in main WARNING workspace.py: 190: File "/home/akhtar/Downloads/Detectron-master/detectron/detectron/utils/train.py", line 54, in train_model WARNING workspace.py: 190: File "/home/akhtar/Downloads/Detectron-master/detectron/detectron/utils/train.py", line 140, in create_model WARNING workspace.py: 190: File "/home/akhtar/Downloads/Detectron-master/detectron/detectron/modeling/model_builder.py", line 124, in create WARNING workspace.py: 190: File "/home/akhtar/Downloads/Detectron-master/detectron/detectron/modeling/model_builder.py", line 89, in generalized_rcnn WARNING workspace.py: 190: File "/home/akhtar/Downloads/Detectron-master/detectron/detectron/modeling/model_builder.py", line 229, in build_generic_detection_model WARNING workspace.py: 190: File "/home/akhtar/Downloads/Detectron-master/detectron/detectron/modeling/optimizer.py", line 40, in build_data_parallel_model WARNING workspace.py: 190: File "/home/akhtar/Downloads/Detectron-master/detectron/detectron/modeling/optimizer.py", line 63, in _build_forward_graph WARNING workspace.py: 190: File "/home/akhtar/Downloads/Detectron-master/detectron/detectron/modeling/model_builder.py", line 189, in _single_gpu_build_func WARNING workspace.py: 190: File "/home/akhtar/Downloads/Detectron-master/detectron/detectron/modeling/rpn_heads.py", line 47, in add_generic_rpn_outputs WARNING workspace.py: 190: File "/home/akhtar/Downloads/Detectron-master/detectron/detectron/modeling/FPN.py", line 469, in add_fpn_rpn_losses Traceback (most recent call last): File "tools/train_net.py", line 128, in main() File "tools/train_net.py", line 110, in main checkpoints = detectron.utils.train.train_model() File "/home/akhtar/Downloads/Detectron-master/detectron/detectron/utils/train.py", line 66, in train_model workspace.RunNet(model.net.Proto().name) File "/usr/local/lib/python2.7/dist-packages/caffe2/python/workspace.py", line 217, in RunNet StringifyNetName(name), num_iter, allow_fail, File "/usr/local/lib/python2.7/dist-packages/caffe2/python/workspace.py", line 178, in CallWithExceptionIntercept return func(*args, **kwargs) RuntimeError: [enforce fail at spatial_narrow_as_op.cu:85] A.dim32(2) >= B.dim32(2). 108 vs 128. Input 0 height must be >= input 1 height. Error from operator: input: "gpu_0/rpn_labels_int32_wide_fpn3" input: "gpu_0/rpn_cls_logits_fpn3" output: "gpu_0/rpn_labels_int32_fpn3" name: "" type: "SpatialNarrowAs" device_option { device_type: 1 cuda_gpu_id: 0 }

ir413 commented 5 years ago

Hi @mamunir, FPN for VGG is not supported at this time. Sorry for the inconvenience. The FPN builder you're using (FPN.add_fpn_VGG) is not part of the Detectron code so it is hard to tell what the issue is without having access to the code.