search

PaddlePaddle / PaddleRS

Awesome Remote Sensing Toolkit based on PaddlePaddle.
Apache License 2.0
365 stars 88 forks source link

代码里的faster_rcnn.py运行失败 #202

Closed cwudis closed 1 week ago

cwudis commented 1 week ago

欢迎您反馈PaddleRS使用问题。辛苦您提供以下信息，以方便我们快速定位和解决问题：

  1. PaddleRS版本： PaddleRS develop
  2. PaddlePaddle版本： PaddlePaddle 2.6.1
  3. 操作系统信息：Windows
  4. Python版本号：3.8
  5. CUDA/cuDNN版本：CUDA12.0/cuDNN 8.9
  6. 完整的代码：原训练脚本faster_rcnn.py
  7. 详细的错误信息与相关log： Traceback (most recent call last): File "tutorials/train/object_detection/faster_rcnn.py", line 75, in model.train( File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\tasks\object_detector.py", line 350, in train return self._real_train(args) File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\tasks\object_detector.py", line 454, in _real_train self.train_loop( File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\tasks\base.py", line 396, in train_loop outputs = self.train_step(step, data, net, optimizer) File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\tasks\base.py", line 705, in train_step outputs = self.run(net, data, mode='train') File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\tasks\object_detector.py", line 157, in run net_out = net(inputs) File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddle\nn\layer\layers.py", line 1429, in call return self.forward(*inputs, *kwargs) File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\models\ppdet\modeling\architectures\meta_arch.py", line 60, in forward out = self.get_loss() File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\models\ppdet\modeling\architectures\faster_rcnn.py", line 112, in get_loss rpn_loss, bbox_loss = self._forward() File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\models\ppdet\modeling\architectures\faster_rcnn.py", line 80, in _forward rois, rois_num, rpn_loss = self.rpn_head(body_feats, self.inputs) File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddle\nn\layer\layers.py", line 1429, in call return self.forward(inputs, kwargs) File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\models\ppdet\modeling\proposal_generator\rpn_head.py", line 141, in forward rois, rois_num = self._gen_proposal(scores, deltas, anchors, inputs) File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\models\ppdet\modeling\proposal_generator\rpn_head.py", line 234, in _gen_proposal bs_rois_num_collect = paddle.concat(bs_rois_num_collect) File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddle\tensor\manipulation.py", line 1263, in concat return _C_ops.concat(input, axis) ValueError: (InvalidArgument) The axis is expected to be in range of [0, 0), but got 0 [Hint: Expected axis >= -rank && axis < rank == true, but received axis >= -rank && axis < rank:0 != true:1.] (at ..\paddle\phi\infermeta\multiary.cc:1035) 运行其他模型没问题
Bobholamovic commented 1 week ago

你好，建议在E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\models\ppdet\modeling\proposal_generator\rpn_head.py的第234行添加断点，检查bs_rois_num_collect中各张量的尺寸。

cwudis commented 1 week ago

image

Bobholamovic commented 1 week ago

这有可能是一个bug。

目前可以尝试在把E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\models\ppdet\modeling\proposal_generator\rpn_head.py的第234行从

bs_rois_num_collect = paddle.concat(bs_rois_num_collect)

改成：

if bs_rois_num_collect[0].ndim >= 1:
    bs_rois_num_collect = paddle.concat(bs_rois_num_collect)
else:
    bs_rois_num_collect = paddle.stack(bs_rois_num_collect)
cwudis commented 1 week ago

改后，报新的错误 Can not use conditional_random_field. Please install pydensecrf first. Warning: import ppdet from source directory without installing, run 'python setup.py install' to install ppdet firstly 2024-06-17 16:18:07,737-WARNING: post-quant-hpo is not support in system other than linux 2024-06-17 16:18:07 [INFO] Decompressing ./data/sarship.zip... 2024-06-17 16:18:11 [DEBUG] ./data/sarship.zip decompressed. 2024-06-17 16:18:11 [INFO] Starting to read file list from dataset... 2024-06-17 16:18:12 [INFO] 175 samples in file ./data/sarship/train.txt, including 175 positive samples and 0 negative samples. creating index... index created! 2024-06-17 16:18:12 [INFO] Starting to read file list from dataset... 2024-06-17 16:18:12 [INFO] 5 samples in file ./data/sarship/eval.txt, including 5 positive samples and 0 negative samples. creating index... index created! W0617 16:18:13.132372 8608 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 8.9, Driver API Version: 12.5, Runtime API Version: 12.0 W0617 16:18:13.139367 8608 gpu_resources.cc:164] device: 0, cuDNN Version: 8.9. 2024-06-17 16:18:13 [WARNING] The initial batch_transforms will be overwritten. 2024-06-17 16:18:13 [INFO] Loading pretrained model from ./output/faster_rcnn/pretrain\faster_rcnn_r50_fpn_2x_coco.pdparams 2024-06-17 16:18:14 [WARNING] [SKIP] Shape of parameters bbox_head.bbox_score.weight do not match. (pretrained: [1024, 81] vs actual: [1024, 2]) 2024-06-17 16:18:14 [WARNING] [SKIP] Shape of parameters bbox_head.bbox_score.bias do not match. (pretrained: [81] vs actual: [2]) 2024-06-17 16:18:14 [WARNING] [SKIP] Shape of parameters bbox_head.bbox_delta.weight do not match. (pretrained: [1024, 320] vs actual: [1024, 4]) 2024-06-17 16:18:14 [WARNING] [SKIP] Shape of parameters bbox_head.bbox_delta.bias do not match. (pretrained: [320] vs actual: [4]) 2024-06-17 16:18:14 [INFO] There are 291/295 variables loaded into FasterRCNN. Traceback (most recent call last): File "tutorials/train/object_detection/faster_rcnn.py", line 75, in model.train( File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\tasks\object_detector.py", line 350, in train return self._real_train(args) File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\tasks\object_detector.py", line 454, in _real_train self.train_loop( File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\tasks\base.py", line 396, in train_loop outputs = self.train_step(step, data, net, optimizer) File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\tasks\base.py", line 705, in train_step outputs = self.run(net, data, mode='train') File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\tasks\object_detector.py", line 157, in run net_out = net(inputs) File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddle\nn\layer\layers.py", line 1429, in call return self.forward(*inputs, *kwargs) File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\models\ppdet\modeling\architectures\meta_arch.py", line 60, in forward out = self.get_loss() File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\models\ppdet\modeling\architectures\faster_rcnn.py", line 112, in get_loss rpn_loss, bbox_loss = self._forward() File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\models\ppdet\modeling\architectures\faster_rcnn.py", line 81, in _forward bboxloss, = self.bbox_head(body_feats, rois, rois_num, File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddle\nn\layer\layers.py", line 1429, in call return self.forward(inputs, kwargs) File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\models\ppdet\modeling\heads\bbox_head.py", line 267, in forward rois, rois_num, targets = self.bbox_assigner(rois, rois_num, inputs) File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\models\ppdet\modeling\proposal_generator\target_layer.py", line 166, in call outs = generate_proposal_target( File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\models\ppdet\modeling\proposal_generator\target.py", line 241, in generate_proposal_target new_rois_num = paddle.concat(new_rois_num) File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddle\tensor\manipulation.py", line 1263, in concat return _C_ops.concat(input, axis) ValueError: (InvalidArgument) The axis is expected to be in range of [0, 0), but got 0 [Hint: Expected axis >= -rank && axis < rank == true, but received axis >= -rank && axis < rank:0 != true:1.] (at ..\paddle\phi\infermeta\multiary.cc:1035)

Bobholamovic commented 1 week ago

看起来是类似的错误，可以尝试对E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\models\ppdet\modeling\proposal_generator\target.py的第241行进行修改，使用类似的方法解决。

cwudis commented 1 week ago

嗯，改完后，可以运行了，但训练不了几轮就报类似的错误，只不过出错位置不同 分别是： Can not use conditional_random_field. Please install pydensecrf first. Warning: import ppdet from source directory without installing, run 'python setup.py install' to install ppdet firstly 2024-06-18 11:46:06,188-WARNING: post-quant-hpo is not support in system other than linux 2024-06-18 11:46:06 [INFO] Decompressing ./data/sarship.zip... 2024-06-18 11:46:10 [DEBUG] ./data/sarship.zip decompressed. 2024-06-18 11:46:10 [INFO] Starting to read file list from dataset... 2024-06-18 11:46:11 [INFO] 175 samples in file ./data/sarship/train.txt, including 175 positive samples and 0 negative samples. creating index... index created! 2024-06-18 11:46:11 [INFO] Starting to read file list from dataset... 2024-06-18 11:46:11 [INFO] 5 samples in file ./data/sarship/eval.txt, including 5 positive samples and 0 negative samples. creating index... index created! W0618 11:46:12.038090 17148 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 8.9, Driver API Version: 12.5, Runtime API Version: 12.0 W0618 11:46:12.045085 17148 gpu_resources.cc:164] device: 0, cuDNN Version: 8.9. 2024-06-18 11:46:12 [WARNING] The initial batch_transforms will be overwritten. 2024-06-18 11:46:12 [INFO] Loading pretrained model from ./output/faster_rcnn/pretrain\faster_rcnn_r50_fpn_2x_coco.pdparams 2024-06-18 11:46:13 [WARNING] [SKIP] Shape of parameters bbox_head.bbox_score.weight do not match. (pretrained: [1024, 81] vs actual: [1024, 2]) 2024-06-18 11:46:13 [WARNING] [SKIP] Shape of parameters bbox_head.bbox_score.bias do not match. (pretrained: [81] vs actual: [2]) 2024-06-18 11:46:13 [WARNING] [SKIP] Shape of parameters bbox_head.bbox_delta.weight do not match. (pretrained: [1024, 320] vs actual: [1024, 4]) 2024-06-18 11:46:13 [WARNING] [SKIP] Shape of parameters bbox_head.bbox_delta.bias do not match. (pretrained: [320] vs actual: [4]) 2024-06-18 11:46:13 [INFO] There are 291/295 variables loaded into FasterRCNN. 2024-06-18 11:46:35 [INFO] [TRAIN] Epoch=1/10, Step=4/43, loss_rpn_cls=0.306890, loss_rpn_reg=0.725881, loss_bbox_cls=0.375581, loss_bbox_reg=0.163465, loss=1.571817, lr=0.005000, time_each_step=5.33s, eta=0:38:2 2024-06-18 11:46:55 [INFO] [TRAIN] Epoch=1/10, Step=8/43, loss_rpn_cls=0.393064, loss_rpn_reg=0.544364, loss_bbox_cls=0.178163, loss_bbox_reg=0.115977, loss=1.231568, lr=0.005000, time_each_step=5.1s, eta=0:36:2 2024-06-18 11:47:16 [INFO] [TRAIN] Epoch=1/10, Step=12/43, loss_rpn_cls=0.246939, loss_rpn_reg=0.390652, loss_bbox_cls=0.234597, loss_bbox_reg=0.231559, loss=1.103747, lr=0.005000, time_each_step=5.29s, eta=0:37:0 2024-06-18 11:47:38 [INFO] [TRAIN] Epoch=1/10, Step=16/43, loss_rpn_cls=0.168808, loss_rpn_reg=0.383643, loss_bbox_cls=0.244856, loss_bbox_reg=0.240769, loss=1.038077, lr=0.005000, time_each_step=5.5s, eta=0:38:8 2024-06-18 11:47:59 [INFO] [TRAIN] Epoch=1/10, Step=20/43, loss_rpn_cls=0.626304, loss_rpn_reg=0.586001, loss_bbox_cls=0.588068, loss_bbox_reg=0.649747, loss=2.450120, lr=0.005000, time_each_step=5.2s, eta=0:35:43 2024-06-18 11:48:19 [INFO] [TRAIN] Epoch=1/10, Step=24/43, loss_rpn_cls=0.257486, loss_rpn_reg=0.390526, loss_bbox_cls=0.201990, loss_bbox_reg=0.291991, loss=1.141993, lr=0.005000, time_each_step=5.02s, eta=0:34:9 2024-06-18 11:48:39 [INFO] [TRAIN] Epoch=1/10, Step=28/43, loss_rpn_cls=0.244560, loss_rpn_reg=0.372828, loss_bbox_cls=0.268240, loss_bbox_reg=0.297336, loss=1.182964, lr=0.005000, time_each_step=4.94s, eta=0:33:16 2024-06-18 11:49:00 [INFO] [TRAIN] Epoch=1/10, Step=32/43, loss_rpn_cls=0.446006, loss_rpn_reg=0.520212, loss_bbox_cls=0.286503, loss_bbox_reg=0.331737, loss=1.584458, lr=0.005000, time_each_step=5.14s, eta=0:34:14 2024-06-18 11:49:21 [INFO] [TRAIN] Epoch=1/10, Step=36/43, loss_rpn_cls=0.259678, loss_rpn_reg=0.430123, loss_bbox_cls=0.117319, loss_bbox_reg=0.106297, loss=0.913417, lr=0.005000, time_each_step=5.31s, eta=0:35:2 2024-06-18 11:49:42 [INFO] [TRAIN] Epoch=1/10, Step=40/43, loss_rpn_cls=0.153984, loss_rpn_reg=0.413964, loss_bbox_cls=0.138056, loss_bbox_reg=0.192680, loss=0.898685, lr=0.005000, time_each_step=5.26s, eta=0:34:23 2024-06-18 11:49:57 [INFO] [TRAIN] Epoch 1 finished, loss_rpn_cls=0.3493410130572874, loss_rpn_reg=0.4661950642286345, loss_bbox_cls=0.2512764199528583, loss_bbox_reg=0.25913346593463144, loss=1.3259459692378377 . 2024-06-18 11:50:03 [INFO] [TRAIN] Epoch=2/10, Step=1/43, loss_rpn_cls=0.455122, loss_rpn_reg=0.452868, loss_bbox_cls=0.131628, loss_bbox_reg=0.190294, loss=1.229912, lr=0.005000, time_each_step=5.36s, eta=0:34:40 2024-06-18 11:50:25 [INFO] [TRAIN] Epoch=2/10, Step=5/43, loss_rpn_cls=0.189669, loss_rpn_reg=0.528992, loss_bbox_cls=0.200498, loss_bbox_reg=0.215080, loss=1.134238, lr=0.005000, time_each_step=5.37s, eta=0:34:20 2024-06-18 11:50:45 [INFO] [TRAIN] Epoch=2/10, Step=9/43, loss_rpn_cls=0.230574, loss_rpn_reg=0.461590, loss_bbox_cls=0.322907, loss_bbox_reg=0.529938, loss=1.545010, lr=0.005000, time_each_step=5.07s, eta=0:32:7 2024-06-18 11:51:06 [INFO] [TRAIN] Epoch=2/10, Step=13/43, loss_rpn_cls=0.307745, loss_rpn_reg=0.373867, loss_bbox_cls=0.213243, loss_bbox_reg=0.360049, loss=1.254905, lr=0.005000, time_each_step=5.26s, eta=0:32:59 2024-06-18 11:51:27 [INFO] [TRAIN] Epoch=2/10, Step=17/43, loss_rpn_cls=0.179184, loss_rpn_reg=0.222601, loss_bbox_cls=0.190243, loss_bbox_reg=0.339382, loss=0.931410, lr=0.005000, time_each_step=5.31s, eta=0:32:56 2024-06-18 11:51:47 [INFO] [TRAIN] Epoch=2/10, Step=21/43, loss_rpn_cls=0.260561, loss_rpn_reg=0.402224, loss_bbox_cls=0.217972, loss_bbox_reg=0.383740, loss=1.264497, lr=0.005000, time_each_step=4.86s, eta=0:29:49 2024-06-18 11:52:08 [INFO] [TRAIN] Epoch=2/10, Step=25/43, loss_rpn_cls=0.195323, loss_rpn_reg=0.383074, loss_bbox_cls=0.196630, loss_bbox_reg=0.331674, loss=1.106701, lr=0.005000, time_each_step=5.34s, eta=0:32:23 2024-06-18 11:52:28 [INFO] [TRAIN] Epoch=2/10, Step=29/43, loss_rpn_cls=0.125539, loss_rpn_reg=0.265280, loss_bbox_cls=0.186745, loss_bbox_reg=0.291602, loss=0.869166, lr=0.005000, time_each_step=4.85s, eta=0:29:7 2024-06-18 11:52:49 [INFO] [TRAIN] Epoch=2/10, Step=33/43, loss_rpn_cls=0.271513, loss_rpn_reg=0.531270, loss_bbox_cls=0.201685, loss_bbox_reg=0.338219, loss=1.342686, lr=0.005000, time_each_step=5.31s, eta=0:31:31 2024-06-18 11:53:10 [INFO] [TRAIN] Epoch=2/10, Step=37/43, loss_rpn_cls=0.220014, loss_rpn_reg=0.404535, loss_bbox_cls=0.212469, loss_bbox_reg=0.372014, loss=1.209032, lr=0.005000, time_each_step=5.35s, eta=0:31:21 2024-06-18 11:53:31 [INFO] [TRAIN] Epoch=2/10, Step=41/43, loss_rpn_cls=0.180509, loss_rpn_reg=0.373099, loss_bbox_cls=0.190887, loss_bbox_reg=0.308534, loss=1.053029, lr=0.005000, time_each_step=5.22s, eta=0:30:15 2024-06-18 11:53:41 [INFO] [TRAIN] Epoch 2 finished, loss_rpn_cls=0.20816632212941036, loss_rpn_reg=0.3524570853211159, loss_bbox_cls=0.19791206474914108, loss_bbox_reg=0.3229728959674059, loss=1.081508366174476 . 2024-06-18 11:53:52 [INFO] [TRAIN] Epoch=3/10, Step=2/43, loss_rpn_cls=0.098014, loss_rpn_reg=0.304489, loss_bbox_cls=0.149317, loss_bbox_reg=0.220986, loss=0.772805, lr=0.005000, time_each_step=5.13s, eta=0:29:23 2024-06-18 11:54:13 [INFO] [TRAIN] Epoch=3/10, Step=6/43, loss_rpn_cls=0.219997, loss_rpn_reg=0.336605, loss_bbox_cls=0.279411, loss_bbox_reg=0.393751, loss=1.229764, lr=0.005000, time_each_step=5.42s, eta=0:30:42 2024-06-18 11:54:34 [INFO] [TRAIN] Epoch=3/10, Step=10/43, loss_rpn_cls=0.175381, loss_rpn_reg=0.463739, loss_bbox_cls=0.207165, loss_bbox_reg=0.233856, loss=1.080141, lr=0.005000, time_each_step=5.12s, eta=0:28:40 2024-06-18 11:54:55 [INFO] [TRAIN] Epoch=3/10, Step=14/43, loss_rpn_cls=0.111875, loss_rpn_reg=0.213160, loss_bbox_cls=0.212481, loss_bbox_reg=0.443002, loss=0.980518, lr=0.005000, time_each_step=5.31s, eta=0:29:22 2024-06-18 11:55:15 [INFO] [TRAIN] Epoch=3/10, Step=18/43, loss_rpn_cls=0.148508, loss_rpn_reg=0.426484, loss_bbox_cls=0.183255, loss_bbox_reg=0.365108, loss=1.123355, lr=0.005000, time_each_step=4.97s, eta=0:27:10 2024-06-18 11:55:37 [INFO] [TRAIN] Epoch=3/10, Step=22/43, loss_rpn_cls=0.185638, loss_rpn_reg=0.436173, loss_bbox_cls=0.241199, loss_bbox_reg=0.302381, loss=1.165392, lr=0.005000, time_each_step=5.46s, eta=0:29:28 2024-06-18 11:55:58 [INFO] [TRAIN] Epoch=3/10, Step=26/43, loss_rpn_cls=0.263669, loss_rpn_reg=0.462009, loss_bbox_cls=0.236246, loss_bbox_reg=0.434107, loss=1.396031, lr=0.005000, time_each_step=5.28s, eta=0:28:9 2024-06-18 11:56:20 [INFO] [TRAIN] Epoch=3/10, Step=30/43, loss_rpn_cls=0.169363, loss_rpn_reg=0.442410, loss_bbox_cls=0.305235, loss_bbox_reg=0.469653, loss=1.386661, lr=0.005000, time_each_step=5.49s, eta=0:28:54 2024-06-18 11:56:42 [INFO] [TRAIN] Epoch=3/10, Step=34/43, loss_rpn_cls=0.089245, loss_rpn_reg=0.214869, loss_bbox_cls=0.179644, loss_bbox_reg=0.377911, loss=0.861670, lr=0.005000, time_each_step=5.55s, eta=0:28:50 2024-06-18 11:57:04 [INFO] [TRAIN] Epoch=3/10, Step=38/43, loss_rpn_cls=0.064340, loss_rpn_reg=0.148108, loss_bbox_cls=0.098305, loss_bbox_reg=0.276777, loss=0.587531, lr=0.005000, time_each_step=5.4s, eta=0:27:43 2024-06-18 11:57:26 [INFO] [TRAIN] Epoch=3/10, Step=42/43, loss_rpn_cls=0.251964, loss_rpn_reg=0.286586, loss_bbox_cls=0.159892, loss_bbox_reg=0.234308, loss=0.932750, lr=0.005000, time_each_step=5.51s, eta=0:27:54 2024-06-18 11:57:30 [INFO] [TRAIN] Epoch 3 finished, loss_rpn_cls=0.15356651624274809, loss_rpn_reg=0.3208635182574738, loss_bbox_cls=0.19191310381473498, loss_bbox_reg=0.32796219753664596, loss=0.9943053459012231 . 2024-06-18 11:57:46 [INFO] [TRAIN] Epoch=4/10, Step=3/43, loss_rpn_cls=0.068906, loss_rpn_reg=0.271396, loss_bbox_cls=0.146419, loss_bbox_reg=0.230301, loss=0.717021, lr=0.005000, time_each_step=5.17s, eta=0:25:50 2024-06-18 11:58:07 [INFO] [TRAIN] Epoch=4/10, Step=7/43, loss_rpn_cls=0.153264, loss_rpn_reg=0.266419, loss_bbox_cls=0.239140, loss_bbox_reg=0.458539, loss=1.117362, lr=0.005000, time_each_step=5.24s, eta=0:25:51 2024-06-18 11:58:30 [INFO] [TRAIN] Epoch=4/10, Step=11/43, loss_rpn_cls=0.098484, loss_rpn_reg=0.313728, loss_bbox_cls=0.115039, loss_bbox_reg=0.129856, loss=0.657107, lr=0.005000, time_each_step=5.58s, eta=0:27:8 2024-06-18 11:58:51 [INFO] [TRAIN] Epoch=4/10, Step=15/43, loss_rpn_cls=0.114065, loss_rpn_reg=0.279468, loss_bbox_cls=0.148531, loss_bbox_reg=0.252175, loss=0.794240, lr=0.005000, time_each_step=5.39s, eta=0:25:51 2024-06-18 11:59:12 [INFO] [TRAIN] Epoch=4/10, Step=19/43, loss_rpn_cls=0.115953, loss_rpn_reg=0.359139, loss_bbox_cls=0.218196, loss_bbox_reg=0.313900, loss=1.007188, lr=0.005000, time_each_step=5.12s, eta=0:24:12 2024-06-18 11:59:33 [INFO] [TRAIN] Epoch=4/10, Step=23/43, loss_rpn_cls=0.180564, loss_rpn_reg=0.339634, loss_bbox_cls=0.190908, loss_bbox_reg=0.353038, loss=1.064144, lr=0.005000, time_each_step=5.28s, eta=0:24:38 2024-06-18 11:59:54 [INFO] [TRAIN] Epoch=4/10, Step=27/43, loss_rpn_cls=0.063062, loss_rpn_reg=0.131187, loss_bbox_cls=0.148998, loss_bbox_reg=0.401446, loss=0.744693, lr=0.005000, time_each_step=5.31s, eta=0:24:25 2024-06-18 12:00:14 [INFO] [TRAIN] Epoch=4/10, Step=31/43, loss_rpn_cls=0.094141, loss_rpn_reg=0.355829, loss_bbox_cls=0.185503, loss_bbox_reg=0.318990, loss=0.954462, lr=0.005000, time_each_step=5.12s, eta=0:23:11 2024-06-18 12:00:36 [INFO] [TRAIN] Epoch=4/10, Step=35/43, loss_rpn_cls=0.138795, loss_rpn_reg=0.408093, loss_bbox_cls=0.210600, loss_bbox_reg=0.330653, loss=1.088141, lr=0.005000, time_each_step=5.45s, eta=0:24:19 2024-06-18 12:00:58 [INFO] [TRAIN] Epoch=4/10, Step=39/43, loss_rpn_cls=0.188317, loss_rpn_reg=0.421094, loss_bbox_cls=0.326682, loss_bbox_reg=0.350248, loss=1.286341, lr=0.005000, time_each_step=5.53s, eta=0:24:18 2024-06-18 12:01:18 [INFO] [TRAIN] Epoch=4/10, Step=43/43, loss_rpn_cls=0.054785, loss_rpn_reg=0.379295, loss_bbox_cls=0.192715, loss_bbox_reg=0.356704, loss=0.983499, lr=0.005000, time_each_step=5.06s, eta=0:21:54 2024-06-18 12:01:18 [INFO] [TRAIN] Epoch 4 finished, loss_rpn_cls=0.12866939093137897, loss_rpn_reg=0.3106158853963364, loss_bbox_cls=0.1751946992305822, loss_bbox_reg=0.3062816501356835, loss=0.9207616379094679 .
2024-06-18 12:01:41 [INFO] [TRAIN] Epoch=5/10, Step=4/43, loss_rpn_cls=0.108889, loss_rpn_reg=0.425793, loss_bbox_cls=0.246288, loss_bbox_reg=0.444199, loss=1.225168, lr=0.005000, time_each_step=5.51s, eta=0:23:31 2024-06-18 12:02:02 [INFO] [TRAIN] Epoch=5/10, Step=8/43, loss_rpn_cls=0.066840, loss_rpn_reg=0.331812, loss_bbox_cls=0.164916, loss_bbox_reg=0.324023, loss=0.887591, lr=0.005000, time_each_step=5.41s, eta=0:22:43 2024-06-18 12:02:23 [INFO] [TRAIN] Epoch=5/10, Step=12/43, loss_rpn_cls=0.073815, loss_rpn_reg=0.238272, loss_bbox_cls=0.162799, loss_bbox_reg=0.366960, loss=0.841847, lr=0.005000, time_each_step=5.24s, eta=0:21:39 2024-06-18 12:02:44 [INFO] [TRAIN] Epoch=5/10, Step=16/43, loss_rpn_cls=0.088353, loss_rpn_reg=0.301441, loss_bbox_cls=0.213698, loss_bbox_reg=0.474088, loss=1.077579, lr=0.005000, time_each_step=5.28s, eta=0:21:28 2024-06-18 12:03:07 [INFO] [TRAIN] Epoch=5/10, Step=20/43, loss_rpn_cls=0.084279, loss_rpn_reg=0.213426, loss_bbox_cls=0.113013, loss_bbox_reg=0.210423, loss=0.621141, lr=0.005000, time_each_step=5.6s, eta=0:22:25 2024-06-18 12:03:28 [INFO] [TRAIN] Epoch=5/10, Step=24/43, loss_rpn_cls=0.076019, loss_rpn_reg=0.277422, loss_bbox_cls=0.136279, loss_bbox_reg=0.218154, loss=0.707873, lr=0.005000, time_each_step=5.44s, eta=0:21:24 2024-06-18 12:03:50 [INFO] [TRAIN] Epoch=5/10, Step=28/43, loss_rpn_cls=0.163400, loss_rpn_reg=0.292277, loss_bbox_cls=0.193181, loss_bbox_reg=0.260862, loss=0.909721, lr=0.005000, time_each_step=5.4s, eta=0:20:52 2024-06-18 12:04:11 [INFO] [TRAIN] Epoch=5/10, Step=32/43, loss_rpn_cls=0.077981, loss_rpn_reg=0.275515, loss_bbox_cls=0.157151, loss_bbox_reg=0.345409, loss=0.856055, lr=0.005000, time_each_step=5.25s, eta=0:19:56 2024-06-18 12:04:31 [INFO] [TRAIN] Epoch=5/10, Step=36/43, loss_rpn_cls=0.110221, loss_rpn_reg=0.269887, loss_bbox_cls=0.224974, loss_bbox_reg=0.412423, loss=1.017505, lr=0.005000, time_each_step=5.09s, eta=0:19:0 2024-06-18 12:04:53 [INFO] [TRAIN] Epoch=5/10, Step=40/43, loss_rpn_cls=0.105802, loss_rpn_reg=0.306461, loss_bbox_cls=0.154323, loss_bbox_reg=0.272000, loss=0.838586, lr=0.005000, time_each_step=5.36s, eta=0:19:38 2024-06-18 12:05:08 [INFO] [TRAIN] Epoch 5 finished, loss_rpn_cls=0.1140497207295063, loss_rpn_reg=0.28902111185151474, loss_bbox_cls=0.1781560981342959, loss_bbox_reg=0.31857951678508933, loss=0.899806446807329 . 2024-06-18 12:05:08 [WARNING] Detector only supports single card evaluation with batch_size=1 during evaluation, so batch_size is forcibly set to 1. 2024-06-18 12:05:09 [INFO] Start to evaluate (total_samples=5, total_steps=5)... Traceback (most recent call last): File "tutorials/train/object_detection/faster_rcnn.py", line 75, in model.train( File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\tasks\object_detector.py", line 350, in train return self._real_train(*args) File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\tasks\object_detector.py", line 454, in _real_train self.train_loop( File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\tasks\base.py", line 449, in train_loop eval_result = self.evaluate( File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\tasks\object_detector.py", line 687, in evaluate outputs = self.run(self.net, data, 'eval') File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\tasks\object_detector.py", line 157, in run File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddle\nn\layer\layers.py", line 1429, in call return self.forward(inputs, **kwargs) File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\models\ppdet\modeling\architectures\meta_arch.py", line 76, in forward outs.append(self.get_pred()) File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\models\ppdet\modeling\architectures\faster_rcnn.py", line 125, in get_pred bbox_pred, bbox_num = self._forward() File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\models\ppdet\modeling\architectures\faster_rcnn.py", line 93, in _forward bboxes, bbox_pred, bbox_num = self.bbox_post_process.get_pred( File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\models\ppdet\modeling\post_process.py", line 134, in get_pred bbox_num = paddle.concat(bbox_num_list) File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddle\tensor\manipulation.py", line 1263, in concat return _C_ops.concat(input, axis) ValueError: (InvalidArgument) The axis is expected to be in range of [0, 0), but got 0 [Hint: Expected axis >= -rank && axis < rank == true, but received axis >= -rank && axis < rank:0 != true:1.] (at ..\paddle\phi\infermeta\multiary.cc:1035)

Traceback (most recent call last): File "tutorials/train/object_detection/faster_rcnn.py", line 75, in model.train( File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\tasks\object_detector.py", line 350, in train return self._real_train(*args) File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\tasks\object_detector.py", line 454, in _real_train self.train_loop( File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\tasks\base.py", line 449, in train_loop eval_result = self.evaluate( File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\tasks\object_detector.py", line 687, in evaluate outputs = self.run(self.net, data, 'eval') File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\tasks\object_detector.py", line 157, in run net_out = net(inputs) File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddle\nn\layer\layers.py", line 1429, in call return self.forward(inputs, **kwargs) File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\models\ppdet\modeling\architectures\meta_arch.py", line 76, in forward outs.append(self.get_pred()) File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\models\ppdet\modeling\architectures\faster_rcnn.py", line 125, in get_pred bbox_pred, bbox_num = self._forward() File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\models\ppdet\modeling\architectures\faster_rcnn.py", line 93, in _forward bboxes, bbox_pred, bbox_num = self.bbox_post_process.get_pred( File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddlers\models\ppdet\modeling\post_process.py", line 150, in get_pred scale = paddle.concat([scale_x, scale_y, scale_x, scale_y]) File "E:\cwudis\software\anaconda3\envs\py3810\lib\site-packages\paddle\tensor\manipulation.py", line 1263, in concat return _C_ops.concat(input, axis) ValueError: (InvalidArgument) The axis is expected to be in range of [0, 0), but got 0 [Hint: Expected axis >= -rank && axis < rank == true, but received axis >= -rank && axis < rank:0 != true:1.] (at ..\paddle\phi\infermeta\multiary.cc:1035)

Bobholamovic commented 1 week ago

类似这样的都是同一种错误，可以尝试根据报错信息，在使用concat的位置按照上面的方式修改～

cwudis commented 1 week ago

好的，谢谢~