distill_pruned_model.py训练报错YOLOv3 output layer number not equal target number

SangerY commented 4 years ago

aistudio上训练了pp-yolo模型作为teacher模型，进行剪枝蒸馏训练，报错YOLOv3 output layer number not equal target number。训练代码：

%cd /home/aistudio/work/PaddleDetection
!python slim/extensions/distill_pruned_model/distill_pruned_model.py \
        -c ./configs/yolov3_mobilenet_v1.yml \
        -t ./configs/ppyolo/ppyolo.yml \
        --teacher_pretrained=output/ppyolo/best_model \
        --pruned_params "yolo_block.0.0.0.conv.weights,yolo_block.0.0.1.conv.weights,yolo_block.0.1.0.conv.weights" \
        --pruned_ratios="0.2,0.3,0.4" \
        -o use_fine_grained_loss=true pretrain_weights=https://paddlemodels.bj.bcebos.com/object_detection/yolov3_mobilenet_v1_voc.tar

baiyfbupt commented 4 years ago

报错YOLOv3 output layer number not equal target number。

您好，麻烦贴下这个报错完整的log信息另外能否分享下aistudio项目，方便我们定位下问题，谢谢~

SangerY commented 4 years ago

@baiyfbupt 您好，完整运行信息如下：

运行耗时: 10秒644毫秒

/home/aistudio/work/PaddleDetection
WARNING: Do not have avx core. You may not build with AVX, but AVX is supported on local machine.
 You could build paddle WITH_AVX=ON to get better performance.
The original error is: No module named 'paddle.fluid.core_avx'
W0907 10:59:36.106340   275 init.cc:185] AVX is available, Please re-compile on local machine
[09-07 10:59:36 MainThread @logger.py:224] Argv: slim/extensions/distill_pruned_model/distill_pruned_model.py -c ./configs/yolov3_mobilenet_v1_gf.yml -t ./configs/ppyolo/ppyolo_gf.yml --teacher_pretrained=output/ppyolo_gf/best_model --pruned_params=yolo_block.0.0.0.conv.weights,yolo_block.0.0.1.conv.weights,yolo_block.0.1.0.conv.weights,yolo_block.0.1.1.conv.weights,yolo_block.0.2.conv.weights,yolo_block.0.tip.conv.weights,yolo_block.1.0.0.conv.weights,yolo_block.1.0.1.conv.weights,yolo_block.1.1.0.conv.weights,yolo_block.1.1.1.conv.weights,yolo_block.1.2.conv.weights,yolo_block.1.tip.conv.weights,yolo_block.2.0.0.conv.weights,yolo_block.2.0.1.conv.weights,yolo_block.2.1.0.conv.weights,yolo_block.2.1.1.conv.weights,yolo_block.2.2.conv.weights,yolo_block.2.tip.conv.weights --pruned_ratios=0.1,0.2,0.2,0.2,0.2,0.1,0.2,0.3,0.3,0.3,0.2,0.1,0.3,0.4,0.4,0.4,0.4,0.3
/opt/conda/envs/python35-paddle120-env/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216, got 192
  return f(*args, **kwds)
/opt/conda/envs/python35-paddle120-env/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/pandas/core/tools/datetimes.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import MutableMapping
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/rcsetup.py:20: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Iterable, Mapping
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/colors.py:53: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Sized
/opt/conda/envs/python35-paddle120-env/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216, got 192
  return f(*args, **kwds)
/opt/conda/envs/python35-paddle120-env/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/framework.py:153: UserWarning: PaddlePaddle version 1.8.0 or higher is required, but 0.0.0 installed, Maybe you are using a develop version, please make sure the version is good with your code.
  (min_version, fluid_version.full_version))
2020-09-07 10:59:36,908-INFO: font search path ['/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/mpl-data/fonts/ttf', '/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/mpl-data/fonts/afm', '/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/mpl-data/fonts/pdfcorefonts']
2020-09-07 10:59:37,244-INFO: generated new fontManager
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
2020-09-07 10:59:42,345-INFO: places would be ommited when DataLoader is not iterable
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
2020-09-07 10:59:42,618-INFO: places would be ommited when DataLoader is not iterable
Traceback (most recent call last):
  File "slim/extensions/distill_pruned_model/distill_pruned_model.py", line 375, in <module>
    main()
  File "slim/extensions/distill_pruned_model/distill_pruned_model.py", line 180, in main
    train_fetches = model.train(teacher_feed_vars)
  File "/home/aistudio/work/PaddleDetection/ppdet/modeling/architectures/yolo.py", line 159, in train
    return self.build(feed_vars, mode='train')
  File "/home/aistudio/work/PaddleDetection/ppdet/modeling/architectures/yolo.py", line 85, in build
    gt_score, targets)
  File "/home/aistudio/work/PaddleDetection/ppdet/modeling/anchor_heads/yolo_head.py", line 401, in get_loss
    self.prefix_name)
  File "/home/aistudio/work/PaddleDetection/ppdet/modeling/losses/yolo_loss.py", line 69, in __call__
    mask_anchors, self._ignore_thresh)
  File "/home/aistudio/work/PaddleDetection/ppdet/modeling/losses/yolo_loss.py", line 128, in _get_fine_grained_loss
    "YOLOv3 output layer number not equal target number"
AssertionError: YOLOv3 output layer number not equal target number

baiyfbupt commented 4 years ago

麻烦贴一下aistudio的链接，我跑一下这个任务debug下

SangerY commented 4 years ago

https://aistudio.baidu.com/aistudio/projectdetail/723018 您看下能打开吗

baiyfbupt commented 4 years ago

运行时碰到的，数据集处理似乎有问题，导致下面又要下载coco数据集，麻烦看下是不是哪里代码弄错了

SangerY commented 4 years ago

@baiyfbupt 你看下data下有没有hot_spot数据，将hot_spot解压到dataset文件夹下，数据集已经生成好了，不需要进行x2coco操作了

baiyfbupt commented 4 years ago

放了还是一样的

baiyfbupt commented 4 years ago

你给我的代码中用的是这个config文件，但是我看了一下你项目中的这个文件确实是读取coco数据的，麻烦确认下？

SangerY commented 4 years ago

@baiyfbupt 以分享的项目里代码为主，官方推荐的两种不同剪枝的我都试了，将use_fine_grained_loss改为false也不行，加了finetune_exclude_pretrained_params也不行：

%cd /home/aistudio/work/PaddleDetection
!python slim/extensions/distill_pruned_model/distill_pruned_model.py \
        -c ./configs/yolov3_mobilenet_v1_gf.yml \
        -t ./configs/ppyolo/ppyolo_gf.yml \
        --teacher_pretrained=output/ppyolo_gf/best_model \
        --pruned_params="yolo_block.0.0.0.conv.weights,yolo_block.0.0.1.conv.weights,yolo_block.0.1.0.conv.weights,yolo_block.0.1.1.conv.weights,yolo_block.0.2.conv.weights,yolo_block.0.tip.conv.weights,yolo_block.1.0.0.conv.weights,yolo_block.1.0.1.conv.weights,yolo_block.1.1.0.conv.weights,yolo_block.1.1.1.conv.weights,yolo_block.1.2.conv.weights,yolo_block.1.tip.conv.weights,yolo_block.2.0.0.conv.weights,yolo_block.2.0.1.conv.weights,yolo_block.2.1.0.conv.weights,yolo_block.2.1.1.conv.weights,yolo_block.2.2.conv.weights,yolo_block.2.tip.conv.weights" \
        --pruned_ratios="0.1,0.2,0.2,0.2,0.2,0.1,0.2,0.3,0.3,0.3,0.2,0.1,0.3,0.4,0.4,0.4,0.4,0.3"  \    
         -o use_fine_grained_loss=true pretrain_weights=https://paddlemodels.bj.bcebos.com/object_detection/yolov3_mobilenet_v1_voc.tar 
        # finetune_exclude_pretrained_params=['cls_score','bbox_pred']

baiyfbupt commented 4 years ago

您好，是这样的，这种剪裁后蒸馏的方法本来是针对yolov3模型开发的，所以像一些蒸馏的对象var name都是取的yolov3模型的，所以使用新模型可以先参照这个文档找到yolov3_loss输入的w,h,x,y,class,objness等variable name，然后替换原代码中的这一部分 https://github.com/PaddlePaddle/PaddleDetection/blob/57a02fd635ad03522b6779b71789846538712a1d/slim/extensions/distill_pruned_model/distill_pruned_model.py#L199-L208

SangerY commented 4 years ago

按照文档打印了student变量，teacher变量没打印出来，程序执行到下面这个地方就报错了。这个teacher变量名怎么获取？

@baiyfbupt 以前用的官方的yolov3_r4作为teacher模型进行yolov3_mobilenetv1蒸馏训练，这次用的pp-yolo作为teacher模型进行yolov3_mobilenetv1剪枝蒸馏训练，打印了student的变量名，模型一样变量不一样？

baiyfbupt commented 4 years ago

我这里可以正常打印出teacher_var，没能复现出你遇到的问题，可能是我们的检测库版本不同？麻烦再分享下你的任务我再试试

以前用的官方的yolov3_r4作为teacher模型进行yolov3_mobilenetv1蒸馏训练，这次用的pp-yolo作为teacher模型进行yolov3_mobilenetv1剪枝蒸馏训练，打印了student的变量名，模型一样变量不一样？

当use_fine_grained_loss为true时，组网会使用拆分后的yolov3_loss，如图左边那样。当它为false使用一个大的yolov3_loss op，如图右边。应该是你两次实验用的use_fine_grained_loss这个属性不一致

SangerY commented 4 years ago

@baiyfbupt https://aistudio.baidu.com/aistudio/projectdetail/723018 版本是v202009231610，我是将打印信息就直接写在了distill_pruned_model.py这个脚本里了

另外，在当前这个项目里，我试了下将use_fine_grained_loss改为true或者false，student变量打印的都是之前图右的变量信息，是我操作上有错误吗？

HelloAI-twj commented 4 years ago

请问楼主问题解决了吗，我也遇到了这个问题，无法打印teacher_var，请问一下您是怎么解决的呢

SangerY commented 4 years ago

请问楼主问题解决了吗，我也遇到了这个问题，无法打印teacher_var，请问一下您是怎么解决的呢

我还没解决呢，后面就没弄这块了。你现在解决了吗

Fauny commented 3 years ago

遇到同样问题，有那位大神能不能帮忙解决啊

PaddleDetection v0.5

qingqing01 commented 3 years ago

@SangerY @Fauny @HelloAI-twj 抱歉，由于问题较久。不知道使用最新的是否有解决？另外，最新2.0版本支持动态图，也支持了剪枝和蒸馏，可以试用

PaddlePaddle / PaddleDetection

distill_pruned_model.py训练报错YOLOv3 output layer number not equal target number #1351