search

PaddlePaddle / PaddleDetection

Object Detection toolkit based on PaddlePaddle. It supports object detection, instance segmentation, multiple object tracking and real-time multi-person keypoint detection.
Apache License 2.0
12.77k stars 2.88k forks source link

在自有数据集上训练mask_rcnn_r50_1x报错valueError #454

Closed Magsun closed 4 years ago

Magsun commented 4 years ago

@百度专家 萌新求助

自己的数据是用labelme标注后使用labelme自带的labelme2coco转换成coco格式数据集，标注了三类数据（不含背景），修改了mask_reader.yml中的data_dir，和drop_last，修改了mask rcnn r50 1x 里num_class=4。

训练时报错广播错误，无法将(3,800,1067)广播到(3,800)。 ValueError: could not broadcast input array from shape (3,800,1067) into shape (3,800)

百度了一下有的人说是输入尺寸有问题，跟模板不一样，我找了找没看见哪里规定了输入尺寸，我看框架本身会对数据进行resize的感觉应该不会存在尺寸问题额。跪求解答QAQ

log如下： (base) root@8bdbc88b7a0b:/workspace/PaddleDetection# python tools/train.py -c configs/mask_rcnn_r50_1x.yml BBoxAssigner: batch_size_per_im: 512 bbox_reg_weights:

W0407 08:11:58.271080 3546 device_context.cc:237] Please NOTE: device: 0, CUDA Capability: 61, Driver API Version: 10.1, Runtime API Version: 10.0 W0407 08:11:58.277384 3546 device_context.cc:245] device: 0, cuDNN Version: 7.6. 2020-04-07 08:12:02,633-INFO: Load model and fuse batch norm if have from https://paddle-imagenet-models-name.bj.bcebos.com/ResNet50_cos_pretrained.tar... 2020-04-07 08:12:02,634-INFO: Found /root/.cache/paddle/weights/ResNet50_cos_pretrained 2020-04-07 08:12:02,641-INFO: Loading parameters from /root/.cache/paddle/weights/ResNet50_cos_pretrained... 2020-04-07 08:12:02,641-WARNING: /root/.cache/paddle/weights/ResNet50_cos_pretrained.pdparams not found, try to load model file saved with [ save_params, save_persistables, save_vars ] 2020-04-07 08:12:02,641-WARNING: /root/.cache/paddle/weights/ResNet50_cos_pretrained.pdparams not found, try to load model file saved with [ save_params, save_persistables, save_vars ] 2020-04-07 08:12:02,650-WARNING: variable file [ /root/.cache/paddle/weights/ResNet50_cos_pretrained/fc_0.w_0 /root/.cache/paddle/weights/ResNet50_cos_pretrained/fc_0.b_0 ] not used 2020-04-07 08:12:02,650-WARNING: variable file [ /root/.cache/paddle/weights/ResNet50_cos_pretrained/fc_0.w_0 /root/.cache/paddle/weights/ResNet50_cos_pretrained/fc_0.b_0 ] not used loading annotations into memory... Done (t=1.09s) creating index... index created! 2020-04-07 08:12:05,022-INFO: 2960 samples in file /data/cloudcover_is/20200331/coco/annotations/instances_train2017.json 2020-04-07 08:12:05,047-INFO: places would be ommited when DataLoader is not iterable I0407 08:12:05.064105 3546 parallel_executor.cc:440] The Program will be executed on CUDA using ParallelExecutor, 1 cards are used, so 1 programs are executed in parallel. I0407 08:12:05.089402 3546 build_strategy.cc:365] SeqOnlyAllReduceOps:0, num_trainers:1 I0407 08:12:05.130786 3546 parallel_executor.cc:307] Inplace strategy is enabled, when build_strategy.enable_inplace = True I0407 08:12:05.149138 3546 parallel_executor.cc:375] Garbage collection strategy is enabled, when FLAGS_eager_delete_tensor_gb = 0 2020-04-07 08:12:07,916-WARNING: Your reader has raised an exception! Exception in thread Thread-6: Traceback (most recent call last): File "/root/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/root/anaconda3/lib/python3.7/threading.py", line 870, in run self._target(*self._args, *self._kwargs) File "/root/anaconda3/lib/python3.7/site-packages/paddle/fluid/reader.py", line 805, in __thread_main__ six.reraise(sys.exc_info()) File "/root/anaconda3/lib/python3.7/site-packages/six.py", line 703, in reraise raise value File "/root/anaconda3/lib/python3.7/site-packages/paddle/fluid/reader.py", line 785, in thread_main for tensors in self._tensor_reader(): File "/root/anaconda3/lib/python3.7/site-packages/paddle/fluid/reader.py", line 853, in tensor_reader_impl for slots in paddle_reader(): File "/root/anaconda3/lib/python3.7/site-packages/paddle/fluid/data_feeder.py", line 489, in reader_creator yield self.feed(item) File "/root/anaconda3/lib/python3.7/site-packages/paddle/fluid/data_feeder.py", line 330, in feed ret_dict[each_name] = each_converter.done() File "/root/anaconda3/lib/python3.7/site-packages/paddle/fluid/data_feeder.py", line 139, in done arr = numpy.array(self.data, dtype=self.dtype) ValueError: could not broadcast input array from shape (3,800,1067) into shape (3,800)

/root/anaconda3/lib/python3.7/site-packages/paddle/fluid/executor.py:782: UserWarning: The following exception is not an EOF exception. "The following exception is not an EOF exception.") Traceback (most recent call last): File "tools/train.py", line 323, in main() File "tools/train.py", line 233, in main outs = exe.run(compiled_train_prog, fetch_list=train_values) File "/root/anaconda3/lib/python3.7/site-packages/paddle/fluid/executor.py", line 783, in run six.reraise(*sys.exc_info()) File "/root/anaconda3/lib/python3.7/site-packages/six.py", line 703, in reraise raise value File "/root/anaconda3/lib/python3.7/site-packages/paddle/fluid/executor.py", line 778, in run use_program_cache=use_program_cache) File "/root/anaconda3/lib/python3.7/site-packages/paddle/fluid/executor.py", line 843, in _run_impl return_numpy=return_numpy) File "/root/anaconda3/lib/python3.7/site-packages/paddle/fluid/executor.py", line 677, in _run_parallel tensors = exe.run(fetch_var_names)._move_to_list() paddle.fluid.core_avx.EnforceNotMet:

C++ Call Stacks (More useful to developers):

0 std::string paddle::platform::GetTraceBackString<std::string const&>(std::string const&, char const, int) 1 paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const, int) 2 paddle::operators::reader::BlockingQueue<std::vector<paddle::framework::LoDTensor, std::allocator > >::Receive(std::vector<paddle::framework::LoDTensor, std::allocator >) 3 paddle::operators::reader::PyReader::ReadNext(std::vector<paddle::framework::LoDTensor, std::allocator >) 4 std::_Function_handler<std::unique_ptr<std::future_base::_Result_base, std::future_base::_Result_base::_Deleter> (), std::future_base::_Task_setter<std::unique_ptr<std::future_base::_Result, std::future_base::_Result_base::_Deleter>, unsigned long> >::_M_invoke(std::_Any_data const&) 5 std::__future_base::_State_base::_M_do_set(std::function<std::unique_ptr<std::future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&) 6 ThreadPool::ThreadPool(unsigned long)::{lambda()#1}::operator()() const

Python Call Stacks (More useful to users):

File "/root/anaconda3/lib/python3.7/site-packages/paddle/fluid/framework.py", line 2525, in append_op attrs=kwargs.get("attrs", None)) File "/root/anaconda3/lib/python3.7/site-packages/paddle/fluid/reader.py", line 733, in _init_non_iterable outputs={'Out': self._feed_list}) File "/root/anaconda3/lib/python3.7/site-packages/paddle/fluid/reader.py", line 646, in init self._init_non_iterable() File "/root/anaconda3/lib/python3.7/site-packages/paddle/fluid/reader.py", line 280, in from_generator iterable, return_list) File "/workspace/PaddleDetection/ppdet/modeling/architectures/mask_rcnn.py", line 329, in build_inputs iterable=iterable) if use_dataloader else None File "tools/train.py", line 115, in main feed_vars, train_loader = model.build_inputs(**inputs_def) File "tools/train.py", line 323, in main()

Error Message Summary:

Error: Blocking queue is killed because the data reader raises an exception [Hint: Expected killed != true, but received killed:1 == true:1.] at (/paddle/paddle/fluid/operators/reader/blocking_queue.h:141) [operator < read > error] terminate called without an active exception W0407 08:12:08.303510 3594 init.cc:209] Warning: PaddlePaddle catches a failure signal, it may not work properly W0407 08:12:08.303535 3594 init.cc:211] You could check whether you killed PaddlePaddle thread/process accidentally or report the case to PaddlePaddle W0407 08:12:08.303544 3594 init.cc:214] The detail failure signal is:

W0407 08:12:08.303556 3594 init.cc:217] Aborted at 1586247128 (unix time) try "date -d @1586247128" if you are using GNU date W0407 08:12:08.306300 3594 init.cc:217] PC: @ 0x0 (unknown) W0407 08:12:08.306440 3594 init.cc:217] SIGABRT (@0xdda) received by PID 3546 (TID 0x7f445d9fd700) from PID 3546; stack trace: W0407 08:12:08.308898 3594 init.cc:217] @ 0x7f454f537390 (unknown) W0407 08:12:08.311234 3594 init.cc:217] @ 0x7f454f191428 gsignal W0407 08:12:08.313705 3594 init.cc:217] @ 0x7f454f19302a abort W0407 08:12:08.317917 3594 init.cc:217] @ 0x7f45267d084a gnu_cxx::verbose_terminate_handler() W0407 08:12:08.319195 3594 init.cc:217] @ 0x7f45267cef47 cxxabiv1::terminate() W0407 08:12:08.320966 3594 init.cc:217] @ 0x7f45267cef7d std::terminate() W0407 08:12:08.322532 3594 init.cc:217] @ 0x7f45267cec5a gxx_personality_v0 W0407 08:12:08.324553 3594 init.cc:217] @ 0x7f454e796b97 _Unwind_ForcedUnwind_Phase2 W0407 08:12:08.326314 3594 init.cc:217] @ 0x7f454e796e7d _Unwind_ForcedUnwind W0407 08:12:08.327750 3594 init.cc:217] @ 0x7f454f536070 GI_pthread_unwind W0407 08:12:08.329143 3594 init.cc:217] @ 0x7f454f52e845 pthread_exit W0407 08:12:08.329684 3594 init.cc:217] @ 0x556edfe321c9 PyThread_exit_thread W0407 08:12:08.329820 3594 init.cc:217] @ 0x556edfcc4cb1 PyEval_RestoreThread.cold.787 W0407 08:12:08.330205 3594 init.cc:217] @ 0x7f450c77bcde (unknown) W0407 08:12:08.330754 3594 init.cc:217] @ 0x556edfdbc114 _PyMethodDef_RawFastCallKeywords W0407 08:12:08.331284 3594 init.cc:217] @ 0x556edfdbc231 _PyCFunction_FastCallKeywords W0407 08:12:08.331825 3594 init.cc:217] @ 0x556edfe20a5d _PyEval_EvalFrameDefault W0407 08:12:08.332321 3594 init.cc:217] @ 0x556edfd756f9 _PyEval_EvalCodeWithName W0407 08:12:08.332814 3594 init.cc:217] @ 0x556edfd76805 _PyFunction_FastCallDict W0407 08:12:08.333302 3594 init.cc:217] @ 0x556edfd91943 _PyObject_Call_Prepend W0407 08:12:08.333562 3594 init.cc:217] @ 0x556edfdd012a slot_tp_call W0407 08:12:08.334058 3594 init.cc:217] @ 0x556edfdd118b _PyObject_FastCallKeywords W0407 08:12:08.334596 3594 init.cc:217] @ 0x556edfe20626 _PyEval_EvalFrameDefault W0407 08:12:08.335088 3594 init.cc:217] @ 0x556edfd7673b _PyFunction_FastCallDict W0407 08:12:08.335577 3594 init.cc:217] @ 0x556edfd91943 _PyObject_Call_Prepend W0407 08:12:08.335862 3594 init.cc:217] @ 0x556edfdd012a slot_tp_call W0407 08:12:08.336364 3594 init.cc:217] @ 0x556edfdd118b _PyObject_FastCallKeywords W0407 08:12:08.336897 3594 init.cc:217] @ 0x556edfe20e8f _PyEval_EvalFrameDefault W0407 08:12:08.337393 3594 init.cc:217] @ 0x556edfd756f9 _PyEval_EvalCodeWithName W0407 08:12:08.337884 3594 init.cc:217] @ 0x556edfd76805 _PyFunction_FastCallDict W0407 08:12:08.338369 3594 init.cc:217] @ 0x556edfd91943 _PyObject_Call_Prepend W0407 08:12:08.338919 3594 init.cc:217] @ 0x556edfd84b9e PyObject_Call Aborted (core dumped)

jerrywgz commented 4 years ago

可以发下配置文件吗

jerrywgz commented 4 years ago

https://github.com/PaddlePaddle/PaddleDetection/blob/release/0.2/ppdet/data/tools/x2coco.py 官方提供了label2coco的数据转换脚本

Magsun commented 4 years ago

可以发下配置文件吗

您好，mask_rcnn_r50_1x.yml配置如下：

architecture: MaskRCNN use_gpu: true max_iters: 180000 snapshot_iter: 1000 log_smooth_window: 20 save_dir: /data/cloudcover_is/20200331/output/mask_rcnn_r50_1x pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNet50_cos_pretrained.tar metric: COCO weights: /data/cloudcover_is/20200331/output/mask_rcnn_r50_1x/model_final num_classes: 4

MaskRCNN: backbone: ResNet rpn_head: RPNHead roi_extractor: RoIAlign bbox_assigner: BBoxAssigner bbox_head: BBoxHead mask_assigner: MaskAssigner mask_head: MaskHead

ResNet: norm_type: affine_channel norm_decay: 0. depth: 50 feature_maps: 4 freeze_at: 2

ResNetC5: depth: 50 norm_type: affine_channel

RPNHead: anchor_generator: anchor_sizes: [32, 64, 128, 256, 512] aspect_ratios: [0.5, 1.0, 2.0] stride: [16.0, 16.0] variance: [1.0, 1.0, 1.0, 1.0] rpn_target_assign: rpn_batch_size_per_im: 256 rpn_fg_fraction: 0.5 rpn_negative_overlap: 0.3 rpn_positive_overlap: 0.7 rpn_straddle_thresh: 0.0 train_proposal: min_size: 0.0 nms_thresh: 0.7 pre_nms_top_n: 12000 post_nms_top_n: 2000 test_proposal: min_size: 0.0 nms_thresh: 0.7 pre_nms_top_n: 6000 post_nms_top_n: 1000

RoIAlign: resolution: 14 spatial_scale: 0.0625 sampling_ratio: 0

BBoxHead: head: ResNetC5 nms: keep_top_k: 100 nms_threshold: 0.5 normalized: false score_threshold: 0.05

MaskHead: dilation: 1 conv_dim: 256 resolution: 14

BBoxAssigner: batch_size_per_im: 512 bbox_reg_weights: [0.1, 0.1, 0.2, 0.2] bg_thresh_hi: 0.5 bg_thresh_lo: 0.0 fg_fraction: 0.25 fg_thresh: 0.5

MaskAssigner: resolution: 14

LearningRate: base_lr: 0.01 schedulers:

OptimizerBuilder: optimizer: momentum: 0.9 type: Momentum regularizer: factor: 0.0001 type: L2

READER: 'mask_reader.yml'

mask_reader.yml配置如下： TrainReader: inputs_def: fields: ['image', 'im_info', 'im_id', 'gt_bbox', 'gt_class', 'is_crowd', 'gt_mask'] dataset: !COCODataSet image_dir: train2017 anno_path: annotations/instances_train2017.json dataset_dir: /data/cloudcover_is/20200331/coco/ sample_transforms:

EvalReader: inputs_def: fields: ['image', 'im_info', 'im_id', 'im_shape']

for voc

#fields: ['image', 'im_info', 'im_id', 'gt_bbox', 'gt_class', 'is_difficult']

dataset: !COCODataSet image_dir: val2017 anno_path: annotations/instances_val2017.json dataset_dir: /data/cloudcover_is/20200331/coco/ sample_transforms:

TestReader: inputs_def: fields: ['image', 'im_info', 'im_id', 'im_shape'] dataset: !ImageFolder anno_path: annotations/instances_val2017.json sample_transforms:

Magsun commented 4 years ago

https://github.com/PaddlePaddle/PaddleDetection/blob/release/0.2/ppdet/data/tools/x2coco.py 官方提供了label2coco的数据转换脚本

正常尝试使用这个脚本重新生成数据集看看是否还是会有一样的报错，感谢！~

qingqing01 commented 4 years ago

关闭此issue，如果还有问题，可以新开issue。