search

PaddlePaddle / PaddleDetection

Object Detection toolkit based on PaddlePaddle. It supports object detection, instance segmentation, multiple object tracking and real-time multi-person keypoint detection.
Apache License 2.0
12.8k stars 2.89k forks source link

敏感度分析无法生成敏感度文件 #5876

Closed personqianduixue closed 2 years ago

personqianduixue commented 2 years ago

描述问题/Describe the bug

按教程做敏感度分析无法生成敏感度文件

复现/Reproduction

运行环境：操作系统win10，python3.7，CUDA10.2/cuDNN7.6.5，paddledetection2.1和2.4，paddleslim2.2.2

敏感度分析教程 https://paddledetection.readthedocs.io/advanced_tutorials/slim/prune/SENSITIVE.html

按敏感度分析教程运行命令如下：

python sensitive.py -c ../../configs/yolov3_mobilenet_v1_roadsign.yml -p "yolo_block.0.0.0.conv.weights,yolo_block.0.0.1.conv.weights,yolo_block.0.1.0.conv.weights,yolo_block.0.1.1.conv.weights,yolo_block.0.2.conv.weights,yolo_block.0.tip.conv.weights,yolo_block.1.0.0.conv.weights,yolo_block.1.0.1.conv.weights,yolo_block.1.1.0.conv.weights,yolo_block.1.1.1.conv.weights,yolo_block.1.2.conv.weights,yolo_block.1.tip.conv.weights,yolo_block.2.0.0.conv.weights,yolo_block.2.0.1.conv.weights,yolo_block.2.1.0.conv.weights,yolo_block.2.1.1.conv.weights,yolo_block.2.2.conv.weights,yolo_block.2.tip.conv.weights" -s "./demo.data"

控制台输出如下：

D:\paddle\PaddleDetection-release-2.4\static\slim\sensitive>python sensitive.py -c ../../configs/yolov3_mobilenet_v1_roadsign.yml -p "yolo_block.0.0.0.conv.weights,yolo_block.0.0.1.conv.weights,yolo_block.0.1.0.conv.weights,yolo_block.0.1.1.conv.weights,yolo_block.0.2.conv.weights,yolo_block.0.tip.conv.weights,yolo_block.1.0.0.conv.weights,yolo_block.1.0.1.conv.weights,yolo_block.1.1.0.conv.weights,yolo_block.1.1.1.conv.weights,yolo_block.1.2.conv.weights,yolo_block.1.tip.conv.weights,yolo_block.2.0.0.conv.weights,yolo_block.2.0.1.conv.weights,yolo_block.2.1.0.conv.weights,yolo_block.2.1.1.conv.weights,yolo_block.2.2.conv.weights,yolo_block.2.tip.conv.weights" -s "./demo.data"
E:\Anaconda\lib\site-packages\win32\lib\pywintypes.py:2: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp, sys, os
FLAGS.config: ../../configs/yolov3_mobilenet_v1_roadsign.yml
2022-05-02 12:57:10,636-INFO: Dataset D:\paddle\PaddleDetection-release-2.4\static\slim\sensitive\dataset\roadsign_voc is not valid for reason above, try searching C:\Users\nancheng/.cache/paddle/dataset or downloading dataset...
2022-05-02 12:57:10,637-INFO: Found C:\Users\nancheng/.cache/paddle/dataset\roadsign_voc\annotations
2022-05-02 12:57:10,638-INFO: Found C:\Users\nancheng/.cache/paddle/dataset\roadsign_voc\images
W0502 12:57:10.704293 19480 device_context.cc:447] Please NOTE: device: 0, GPU Compute Capability: 7.5, Driver API Version: 11.4, Runtime API Version: 10.2
W0502 12:57:10.712293 19480 device_context.cc:465] device: 0, cuDNN Version: 7.6.
2022-05-02 12:57:14,239-INFO: pruned params: ['yolo_block.0.0.0.conv.weights', 'yolo_block.0.0.1.conv.weights', 'yolo_block.0.1.0.conv.weights', 'yolo_block.0.1.1.conv.weights', 'yolo_block.0.2.conv.weights', 'yolo_block.0.tip.conv.weights', 'yolo_block.1.0.0.conv.weights', 'yolo_block.1.0.1.conv.weights', 'yolo_block.1.1.0.conv.weights', 'yolo_block.1.1.1.conv.weights', 'yolo_block.1.2.conv.weights', 'yolo_block.1.tip.conv.weights', 'yolo_block.2.0.0.conv.weights', 'yolo_block.2.0.1.conv.weights', 'yolo_block.2.1.0.conv.weights', 'yolo_block.2.1.1.conv.weights', 'yolo_block.2.2.conv.weights', 'yolo_block.2.tip.conv.weights']
2022-05-02 12:57:14,239-INFO: pruned ratios: [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]
2022-05-02 12:57:14,457-INFO: Test iter 0
2022-05-02 12:57:19,071-INFO: Test iter 100

这样就结束了

查看当前目录没有生成demo.data敏感度文件，试了paddledetection2.1和2.4版本都不行，请问打扰如何解决？

moneypi commented 2 years ago

Xubuntu 20.04 python 3.9.7

(base) luke@luke-NH5x-7xRCx-RDx:~/PaddleDetection/static/slim/sensitive$ pip list|grep paddle
paddle-upgrade-tool           0.0.27
paddle2onnx                   0.9.5
paddlefsl                     1.1.0
paddlehub                     2.2.0
paddlenlp                     2.2.6
paddlepaddle-gpu              2.2.2
paddleslim                    2.2.2
paddleslim-opt-tools          2.11
(base) luke@luke-NH5x-7xRCx-RDx:~/PaddleDetection/static/slim/sensitive$ python sensitive.py -c ../../configs/yolov3_mobilenet_v1_voc.yml --pruned_params "yolo_block.0.0.0.conv.weights,yolo_block.0.0.1.conv.weights,yolo_block.0.1.0.conv.weights,yolo_block.0.1.1.conv.weights,yolo_block.0.2.conv.weights,yolo_block.0.tip.conv.weights,yolo_block.1.0.0.conv.weights,yolo_block.1.0.1.conv.weights,yolo_block.1.1.0.conv.weights,yolo_block.1.1.1.conv.weights,yolo_block.1.2.conv.weights,yolo_block.1.tip.conv.weights,yolo_block.2.0.0.conv.weights,yolo_block.2.0.1.conv.weights,yolo_block.2.1.0.conv.weights,yolo_block.2.1.1.conv.weights,yolo_block.2.2.conv.weights,yolo_block.2.tip.conv.weights" --sensitivities_file "./demo.data"
/home/luke/miniconda3/lib/python3.9/site-packages/fontTools/misc/py23.py:11: DeprecationWarning: The py23 module has been deprecated and will be removed in a future release. Please update your code.
  warnings.warn(
FLAGS.config: ../../configs/yolov3_mobilenet_v1_voc.yml
2022-05-03 23:55:54,378-INFO: Dataset /home/luke/PaddleDetection/static/slim/sensitive/dataset/voc is not valid for reason above, try searching /home/luke/.cache/paddle/dataset or downloading dataset...
2022-05-03 23:55:54,380-INFO: Found /home/luke/.cache/paddle/dataset/voc/VOCdevkit/VOC2012
2022-05-03 23:55:54,380-INFO: Found /home/luke/.cache/paddle/dataset/voc/VOCdevkit/VOC2007
W0503 23:55:54.836243 72799 device_context.cc:447] Please NOTE: device: 0, GPU Compute Capability: 7.5, Driver API Version: 11.6, Runtime API Version: 10.2
W0503 23:55:54.839100 72799 device_context.cc:465] device: 0, cuDNN Version: 7.6.
2022-05-03 23:55:56,749-INFO: pruned params: ['yolo_block.0.0.0.conv.weights', 'yolo_block.0.0.1.conv.weights', 'yolo_block.0.1.0.conv.weights', 'yolo_block.0.1.1.conv.weights', 'yolo_block.0.2.conv.weights', 'yolo_block.0.tip.conv.weights', 'yolo_block.1.0.0.conv.weights', 'yolo_block.1.0.1.conv.weights', 'yolo_block.1.1.0.conv.weights', 'yolo_block.1.1.1.conv.weights', 'yolo_block.1.2.conv.weights', 'yolo_block.1.tip.conv.weights', 'yolo_block.2.0.0.conv.weights', 'yolo_block.2.0.1.conv.weights', 'yolo_block.2.1.0.conv.weights', 'yolo_block.2.1.1.conv.weights', 'yolo_block.2.2.conv.weights', 'yolo_block.2.tip.conv.weights']
2022-05-03 23:55:56,749-INFO: pruned ratios: [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]
2022-05-03 23:55:57,477-INFO: Test iter 0
2022-05-03 23:56:09,542-INFO: Test iter 100
2022-05-03 23:56:21,262-INFO: Test iter 200
2022-05-03 23:56:33,167-INFO: Test iter 300
2022-05-03 23:56:45,106-INFO: Test iter 400
2022-05-03 23:56:57,378-INFO: Test iter 500
2022-05-03 23:57:09,327-INFO: Test iter 600
2022-05-03 23:57:11,517-INFO: Test finish iter 619
2022-05-03 23:57:11,517-INFO: Total number of images: 4952, inference time: 66.23463804161312 fps.
2022-05-03 23:57:11,518-INFO: Start evaluate...
2022-05-03 23:57:12,929-INFO: Accumulating evaluatation results...
2022-05-03 23:57:13,008-INFO: mAP(0.50, 11point) = 51.78%
2022-05-03 23:57:13,029-INFO: sensitive - param: yolo_block.0.0.0.conv.weights; ratios: 0.1
Traceback (most recent call last):
  File "/home/luke/PaddleDetection/static/slim/sensitive/sensitive.py", line 203, in <module>
    main()
  File "/home/luke/PaddleDetection/static/slim/sensitive/sensitive.py", line 154, in main
    sensitivity(
  File "/home/luke/miniconda3/lib/python3.9/site-packages/paddleslim/prune/sensitive.py", line 95, in sensitivity
    pruned_program, param_backup, _ = pruner.prune(
  File "/home/luke/miniconda3/lib/python3.9/site-packages/paddleslim/prune/pruner.py", line 88, in prune
    collections = StaticPruningCollections(params, graph)
  File "/home/luke/miniconda3/lib/python3.9/site-packages/paddleslim/prune/collections.py", line 243, in __init__
    self._collections = self.create_pruning_collections(
  File "/home/luke/miniconda3/lib/python3.9/site-packages/paddleslim/prune/collections.py", line 182, in create_pruning_collections
    leaves = self._find_leaves(graph)
  File "/home/luke/miniconda3/lib/python3.9/site-packages/paddleslim/prune/collections.py", line 136, in _find_leaves
    if len(_var.outputs()) == 0:
  File "/home/luke/miniconda3/lib/python3.9/site-packages/paddleslim/core/graph_wrapper.py", line 103, in outputs
    if self in op.all_inputs():
  File "/home/luke/miniconda3/lib/python3.9/site-packages/paddleslim/core/graph_wrapper.py", line 67, in __eq__
    return self._var.name == other._var.name
AttributeError: 'NoneType' object has no attribute '_var'
(base) luke@luke-NH5x-7xRCx-RDx:~/PaddleDetection/static/slim/sensitive$ pip install paddledet
Collecting paddledet
  Using cached paddledet-2.4.0-py3-none-any.whl (641 kB)
Requirement already satisfied: setuptools>=42.0.0 in /home/luke/miniconda3/lib/python3.9/site-packages (from paddledet) (58.0.4)
Requirement already satisfied: scipy in /home/luke/miniconda3/lib/python3.9/site-packages (from paddledet) (1.7.3)
Requirement already satisfied: motmetrics in /home/luke/miniconda3/lib/python3.9/site-packages (from paddledet) (1.2.5)
Requirement already satisfied: Cython in /home/luke/miniconda3/lib/python3.9/site-packages (from paddledet) (0.29.28)
Requirement already satisfied: lap in /home/luke/miniconda3/lib/python3.9/site-packages (from paddledet) (0.4.0)
Requirement already satisfied: terminaltables in /home/luke/miniconda3/lib/python3.9/site-packages (from paddledet) (3.1.10)
Requirement already satisfied: pycocotools in /home/luke/miniconda3/lib/python3.9/site-packages (from paddledet) (2.0.4)
Requirement already satisfied: openpyxl in /home/luke/miniconda3/lib/python3.9/site-packages (from paddledet) (3.0.9)
Requirement already satisfied: PyYAML in /home/luke/miniconda3/lib/python3.9/site-packages (from paddledet) (6.0)
Requirement already satisfied: tqdm in /home/luke/miniconda3/lib/python3.9/site-packages (from paddledet) (4.62.3)
Requirement already satisfied: typeguard in /home/luke/miniconda3/lib/python3.9/site-packages (from paddledet) (2.13.3)
Requirement already satisfied: sklearn in /home/luke/miniconda3/lib/python3.9/site-packages (from paddledet) (0.0)
Requirement already satisfied: opencv-python in /home/luke/miniconda3/lib/python3.9/site-packages (from paddledet) (4.5.5.64)
Requirement already satisfied: shapely in /home/luke/miniconda3/lib/python3.9/site-packages (from paddledet) (1.8.1.post1)
Requirement already satisfied: numpy>=1.12.1 in /home/luke/miniconda3/lib/python3.9/site-packages (from motmetrics->paddledet) (1.21.5)
Requirement already satisfied: xmltodict>=0.12.0 in /home/luke/miniconda3/lib/python3.9/site-packages (from motmetrics->paddledet) (0.12.0)
Requirement already satisfied: pandas>=0.23.1 in /home/luke/miniconda3/lib/python3.9/site-packages (from motmetrics->paddledet) (1.4.2)
Requirement already satisfied: pytz>=2020.1 in /home/luke/miniconda3/lib/python3.9/site-packages (from pandas>=0.23.1->motmetrics->paddledet) (2021.3)
Requirement already satisfied: python-dateutil>=2.8.1 in /home/luke/miniconda3/lib/python3.9/site-packages (from pandas>=0.23.1->motmetrics->paddledet) (2.8.2)
Requirement already satisfied: six>=1.5 in /home/luke/miniconda3/lib/python3.9/site-packages (from python-dateutil>=2.8.1->pandas>=0.23.1->motmetrics->paddledet) (1.16.0)
Requirement already satisfied: et-xmlfile in /home/luke/miniconda3/lib/python3.9/site-packages (from openpyxl->paddledet) (1.1.0)
Requirement already satisfied: matplotlib>=2.1.0 in /home/luke/miniconda3/lib/python3.9/site-packages (from pycocotools->paddledet) (3.5.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /home/luke/miniconda3/lib/python3.9/site-packages (from matplotlib>=2.1.0->pycocotools->paddledet) (1.3.2)
Requirement already satisfied: cycler>=0.10 in /home/luke/miniconda3/lib/python3.9/site-packages (from matplotlib>=2.1.0->pycocotools->paddledet) (0.11.0)
Requirement already satisfied: pillow>=6.2.0 in /home/luke/miniconda3/lib/python3.9/site-packages (from matplotlib>=2.1.0->pycocotools->paddledet) (9.1.0)
Requirement already satisfied: pyparsing>=2.2.1 in /home/luke/miniconda3/lib/python3.9/site-packages (from matplotlib>=2.1.0->pycocotools->paddledet) (3.0.4)
Requirement already satisfied: fonttools>=4.22.0 in /home/luke/miniconda3/lib/python3.9/site-packages (from matplotlib>=2.1.0->pycocotools->paddledet) (4.25.0)
Requirement already satisfied: packaging>=20.0 in /home/luke/miniconda3/lib/python3.9/site-packages (from matplotlib>=2.1.0->pycocotools->paddledet) (21.3)
Requirement already satisfied: scikit-learn in /home/luke/miniconda3/lib/python3.9/site-packages (from sklearn->paddledet) (1.0.2)
Requirement already satisfied: joblib>=0.11 in /home/luke/miniconda3/lib/python3.9/site-packages (from scikit-learn->sklearn->paddledet) (1.1.0)
Requirement already satisfied: threadpoolctl>=2.0.0 in /home/luke/miniconda3/lib/python3.9/site-packages (from scikit-learn->sklearn->paddledet) (2.2.0)
Installing collected packages: paddledet
Successfully installed paddledet-2.4.0
(base) luke@luke-NH5x-7xRCx-RDx:~/PaddleDetection/static/slim/sensitive$ python sensitive.py -c ../../configs/yolov3_mobilenet_v1_voc.yml --pruned_params "yolo_block.0.0.0.conv.weights,yolo_block.0.0.1.conv.weights,yolo_block.0.1.0.conv.weights,yolo_block.0.1.1.conv.weights,yolo_block.0.2.conv.weights,yolo_block.0.tip.conv.weights,yolo_block.1.0.0.conv.weights,yolo_block.1.0.1.conv.weights,yolo_block.1.1.0.conv.weights,yolo_block.1.1.1.conv.weights,yolo_block.1.2.conv.weights,yolo_block.1.tip.conv.weights,yolo_block.2.0.0.conv.weights,yolo_block.2.0.1.conv.weights,yolo_block.2.1.0.conv.weights,yolo_block.2.1.1.conv.weights,yolo_block.2.2.conv.weights,yolo_block.2.tip.conv.weights" --sensitivities_file "./demo.data"
/home/luke/miniconda3/lib/python3.9/site-packages/paddle/tensor/creation.py:130: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe. 
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  if data.dtype == np.object:
Traceback (most recent call last):
  File "/home/luke/PaddleDetection/static/slim/sensitive/sensitive.py", line 30, in <module>
    from ppdet.data.reader import create_reader
ImportError: cannot import name 'create_reader' from 'ppdet.data.reader' (/home/luke/miniconda3/lib/python3.9/site-packages/ppdet/data/reader.py)
moneypi commented 2 years ago 
(base) luke@luke-NH5x-7xRCx-RDx:~/PaddleDetection/static$ python slim/prune/prune.py \
> -c ./configs/yolov3_mobilenet_v1_voc.yml \
> --pruned_params "yolo_block.0.0.0.conv.weights,yolo_block.0.0.1.conv.weights,yolo_block.0.1.0.conv.weights" \
> --pruned_ratios="0.2,0.3,0.4"
/home/luke/miniconda3/lib/python3.9/site-packages/fontTools/misc/py23.py:11: DeprecationWarning: The py23 module has been deprecated and will be removed in a future release. Please update your code.
  warnings.warn(
/home/luke/miniconda3/lib/python3.9/site-packages/paddle/tensor/creation.py:130: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe. 
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  if data.dtype == np.object:
Traceback (most recent call last):
  File "/home/luke/PaddleDetection/static/slim/prune/prune.py", line 61, in <module>
    raise e
  File "/home/luke/PaddleDetection/static/slim/prune/prune.py", line 41, in <module>
    from ppdet.experimental import mixed_precision_context
ModuleNotFoundError: No module named 'ppdet.experimental'
moneypi commented 2 years ago 
(base) luke@luke-NH5x-7xRCx-RDx:~/PaddleDetection/static$ pip uninstall paddledet
Found existing installation: paddledet 2.4.0
Uninstalling paddledet-2.4.0:
  Would remove:
    /home/luke/miniconda3/lib/python3.9/site-packages/paddledet-2.4.0.dist-info/*
    /home/luke/miniconda3/lib/python3.9/site-packages/ppdet/*
Proceed (Y/n)? y
  Successfully uninstalled paddledet-2.4.0
(base) luke@luke-NH5x-7xRCx-RDx:~/PaddleDetection/static$ python slim/prune/prune.py -c ./configs/yolov3_mobilenet_v1_voc.yml --pruned_params "yolo_block.0.0.0.conv.weights,yolo_block.0.0.1.conv.weights,yolo_block.0.1.0.conv.weights" --pruned_ratios="0.2,0.3,0.4"
/home/luke/miniconda3/lib/python3.9/site-packages/fontTools/misc/py23.py:11: DeprecationWarning: The py23 module has been deprecated and will be removed in a future release. Please update your code.
  warnings.warn(
/home/luke/miniconda3/lib/python3.9/site-packages/paddle/fluid/framework.py:744: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  elif dtype == np.bool:
2022-05-04 00:40:27,784-INFO: If regularizer of a Parameter has been set by 'fluid.ParamAttr' or 'fluid.WeightNormParamAttr' already. The Regularization[L2Decay, regularization_coeff=0.000500] in Optimizer will not take effect, and it will only be applied to other Parameters!
W0504 00:40:27.943181 81260 device_context.cc:447] Please NOTE: device: 0, GPU Compute Capability: 7.5, Driver API Version: 11.6, Runtime API Version: 10.2
W0504 00:40:27.954591 81260 device_context.cc:465] device: 0, cuDNN Version: 7.6.
/home/luke/miniconda3/lib/python3.9/site-packages/paddle/fluid/io.py:2407: UserWarning: This list is not set, Because of Paramerter not found in program. There are: fc7_offset fc7_weights
  warnings.warn(
2022-05-04 00:40:30,183-INFO: pruned params: ['yolo_block.0.0.0.conv.weights', 'yolo_block.0.0.1.conv.weights', 'yolo_block.0.1.0.conv.weights']
2022-05-04 00:40:30,183-INFO: pruned ratios: [0.2, 0.3, 0.4]
Traceback (most recent call last):
  File "/home/luke/PaddleDetection/static/slim/prune/prune.py", line 427, in <module>
    main()
  File "/home/luke/PaddleDetection/static/slim/prune/prune.py", line 221, in main
    train_prog = pruner.prune(
  File "/home/luke/miniconda3/lib/python3.9/site-packages/paddleslim/prune/pruner.py", line 88, in prune
    collections = StaticPruningCollections(params, graph)
  File "/home/luke/miniconda3/lib/python3.9/site-packages/paddleslim/prune/collections.py", line 243, in __init__
    self._collections = self.create_pruning_collections(
  File "/home/luke/miniconda3/lib/python3.9/site-packages/paddleslim/prune/collections.py", line 182, in create_pruning_collections
    leaves = self._find_leaves(graph)
  File "/home/luke/miniconda3/lib/python3.9/site-packages/paddleslim/prune/collections.py", line 136, in _find_leaves
    if len(_var.outputs()) == 0:
  File "/home/luke/miniconda3/lib/python3.9/site-packages/paddleslim/core/graph_wrapper.py", line 103, in outputs
    if self in op.all_inputs():
  File "/home/luke/miniconda3/lib/python3.9/site-packages/paddleslim/core/graph_wrapper.py", line 67, in __eq__
    return self._var.name == other._var.name
AttributeError: 'NoneType' object has no attribute '_var'
moneypi commented 2 years ago

fix my problem by downgrade paddleslim to 2.1.0

(base) luke@luke-NH5x-7xRCx-RDx:~/PaddleDetection/static$ pip list|grep paddle
paddle-upgrade-tool           0.0.27
paddle2onnx                   0.9.5
paddlefsl                     1.1.0
paddlehub                     2.2.0
paddlenlp                     2.2.6
paddlepaddle-gpu              2.2.2
paddleslim                    2.1.0
paddleslim-opt-tools          2.11
(base) luke@luke-NH5x-7xRCx-RDx:~/PaddleDetection/static$

run sensitive.py for 3.5 hours, and run prune.py for 2.2 hours