search

PaddlePaddle / PaddleDetection

Object Detection toolkit based on PaddlePaddle. It supports object detection, instance segmentation, multiple object tracking and real-time multi-person keypoint detection.
Apache License 2.0
12.77k stars 2.88k forks source link

Paddle3.0beta联合paddledetection2.7进行模型训练时报错 #9058

Open carter275 opened 3 months ago

carter275 commented 3 months ago

问题确认 Search before asking

请提出你的问题 Please ask your question

Warning: Unable to use numba in PP-Tracking, please install numba, for example(python3.7): pip install numba==0.56.4 Warning: Unable to use numba in PP-Tracking, please install numba, for example(python3.7): pip install numba==0.56.4 Warning: import ppdet from source directory without installing, run 'python setup.py install' to install ppdet firstly ======================= Modified FLAGS detected ======================= FLAGS(name='FLAGS_cublas_dir', current_value='/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/../nvidia/cublas/lib', default_value='') FLAGS(name='FLAGS_nccl_dir', current_value='/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/../nvidia/nccl/lib', default_value='') FLAGS(name='FLAGS_nvidia_package_dir', current_value='/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/../nvidia', default_value='') FLAGS(name='FLAGS_curand_dir', current_value='/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/../nvidia/curand/lib', default_value='') FLAGS(name='FLAGS_cusparse_dir', current_value='/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/../nvidia/cusparse/lib', default_value='') FLAGS(name='FLAGS_cusolver_dir', current_value='/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/../nvidia/cusolver/lib', default_value='') FLAGS(name='FLAGS_cudnn_dir', current_value='/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/../nvidia/cudnn/lib', default_value='') FLAGS(name='FLAGS_cupti_dir', current_value='/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/../nvidia/cuda_cupti/lib', default_value='')

W0714 17:39:17.975699 85915 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 12.0, Runtime API Version: 11.8 W0714 17:39:17.977880 85915 gpu_resources.cc:164] device: 0, cuDNN Version: 8.9. [07/14 17:39:31] ppdet.utils.checkpoint INFO: Finish loading model weights: /home/aistudio/.cache/paddle/weights/LCNet_x1_5_pretrained.pdparams Traceback (most recent call last): File "/home/aistudio/PaddleDetection-release-2.7/tools/train.py", line 209, in main() File "/home/aistudio/PaddleDetection-release-2.7/tools/train.py", line 205, in main run(FLAGS, cfg) File "/home/aistudio/PaddleDetection-release-2.7/tools/train.py", line 158, in run trainer.train(FLAGS.eval) File "/home/aistudio/PaddleDetection-release-2.7/ppdet/engine/trainer.py", line 577, in train outputs = model(data) File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1426, in call return self.forward(*inputs, kwargs) File "/home/aistudio/PaddleDetection-release-2.7/ppdet/modeling/architectures/meta_arch.py", line 60, in forward out = self.get_loss() File "/home/aistudio/PaddleDetection-release-2.7/ppdet/modeling/architectures/picodet.py", line 83, in get_loss loss_gfl = self.head.get_loss(head_outs, self.inputs) File "/home/aistudio/PaddleDetection-release-2.7/ppdet/modeling/heads/pico_head.py", line 664, in get_loss assigned_labels, assigned_bboxes, assigned_scores = self.static_assigner( File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1426, in call return self.forward(*inputs, *kwargs) File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/decorator.py", line 232, in fun return caller(func, (extras + args), kw) File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/base/dygraph/base.py", line 337, in _decorate_function return func(*args, *kwargs) File "/home/aistudio/PaddleDetection-release-2.7/ppdet/modeling/assigners/atss_assigner.py", line 195, in forward assigned_gt_index = assigned_gt_index + batch_ind num_max_boxes TypeError: (InvalidType) Type promotion only support calculations between floating-point numbers and between complex and real numbers. But got different data type x: int64, y: int32. (at ../paddle/phi/common/type_promotion.h:164)

etaixiee commented 3 months ago

我这边也遇到相同的问题mark

BotAndyGao commented 2 months ago

建议降低Paddle的版本

这套搭配可以跑起来，亲测多个模型均可以完成训练，识别精度都不错。

yski commented 3 weeks ago

建议降低Paddle的版本

  • Python: 3.10.12
  • CUDA Version: 12.1
  • Paddle版本：2.5.2， 安装方式：Docker
  • PaddleDetection版本： release/2.7， 安装方式：源码

这套搭配可以跑起来，亲测多个模型均可以完成训练，识别精度都不错。 你好，试过MASK-RTDETR吗？我导出模型使用trt加载报错了，也在怀疑是不是paddle的版本问题

liuwake commented 3 weeks ago

I add assigned_gt_index = paddle.cast(assigned_gt_index, dtype="int32") before this line of code assigned_gt_index = assigned_gt_index + batch_ind * num_max_boxes. It works for paddle3.0.0b1 both on Ubuntu 22.04 cu188 RTX4090 and WSL2 Ubuntu 24 cu188 RTX4090.

zzwtop1 commented 1 week ago

I add assigned_gt_index = paddle.cast(assigned_gt_index, dtype="int32") before this line of code assigned_gt_index = assigned_gt_index + batch_ind * num_max_boxes. It works for paddle3.0.0b1 both on Ubuntu 22.04 cu188 RTX4090 and WSL2 Ubuntu 24 cu188 RTX4090.

这个方法解决了我的同类型报错