Closed SarBH closed 2 years ago
Hi @SarBH , Thanks for reporting and providing this information!
All else in the models remained the same, therefore I wondered if this was accidental.
The origin of this change is in upstream YOLOv5 https://github.com/ultralytics/yolov5/pull/4420, the SPPF
we adopted here is a faster version of SPP
. I think it will also benefit the earlier version, as such I replace this part in https://github.com/zhiqwang/yolov5-rt-stack/pull/194/commits/451f3e4be72700ab08c8a1617a5ef4cfc3d0869a with a verification of numerical equality.
EDITED: A detailed test about the SPPF vs SPP - https://github.com/ultralytics/yolov5/pull/4420#issuecomment-995474540 .
I'm using
nni==2.4
to prune the model, and a node with three outputs is a problem for that
Actually, I don't quite understand the phenomenon that occurs here, could you provide me with more information, or a reproducible example? (I also have some interest in nni
and am following its progress).
If the above change affects the use of downstream application, we can revert this change back or we can work together to find a better way to handle this scenario.
Thanks for the follow up @zhiqwang, and for this awesome project!
I investigated where exactly pruning is failing:
the compressor.py file asserts that assert len(node.outputs) == 1, 'The number of the output should be one after the Tuple unpacked manually'
.
I see that all other (non failing) nodes in this model indeed have a single output, but this one MaxPool2d
node has 3 outputs after the update (Below is a printout of the node, see last line):
name: backbone.body.8.m, type: module, op_type: MaxPool2d, sub_nodes:
['__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct',
'__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_aten::max_pool2d',
'__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct',
'__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct',
'__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct',
'__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_aten::max_pool2d',
'__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct',
'__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct',
'__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct',
'__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct',
'__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_aten::max_pool2d',
'__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct',
'__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct',
'__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct',
'__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct'],
inputs: ['input.85'], outputs: ['input.86', 'input.87', '4815'], aux: None
The old hash’s model that doesnt fail prunning actually splits that backbone.body.8.m
layer into backbone.body.8.m.0, backbone.body.8.m.1, backbone.body.8.m.2
, but it is otherwise the same exact model.
** Note: We are using an older version of nni package nni==2.4
so I'm not sure if this is resolved already. From the fact that compressor.py still has the assert im inclined to beleive new version will also fail.
I'm not an expert on pruning, but I hope this helps :)
I see that all other (non failing) nodes in this model indeed have a single output, but this one MaxPool2d node has 3 outputs after the update.
Got it, Thanks for the details informations, it is very useful. I agree with you, we will revert this substitution it this two days.
Hi @SarBH ,
I revert the SPPF
to SPP
in #240 both in "r4.0" and "r6.0", and as such I'm closing this issue.
Please reinstall the yolort
from source (we'll distribute 0.6.0 at the end of the month.)
pip install -U 'git+https://github.com/zhiqwang/yolov5-rt-stack.git'
Thanks for the detailed information again, feel free to reopen this or create another ticket if you have more questions.
Thanks for the follow up @zhiqwang, and for this awesome project!
I investigated where exactly pruning is failing:
the compressor.py file asserts that
assert len(node.outputs) == 1, 'The number of the output should be one after the Tuple unpacked manually'
. I see that all other (non failing) nodes in this model indeed have a single output, but this oneMaxPool2d
node has 3 outputs after the update (Below is a printout of the node, see last line):name: backbone.body.8.m, type: module, op_type: MaxPool2d, sub_nodes: ['__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct', '__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_aten::max_pool2d', '__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct', '__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct', '__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct', '__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_aten::max_pool2d', '__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct', '__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct', '__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct', '__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct', '__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_aten::max_pool2d', '__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct', '__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct', '__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct', '__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct'], inputs: ['input.85'], outputs: ['input.86', 'input.87', '4815'], aux: None
The old hash’s model that doesnt fail prunning actually splits that
backbone.body.8.m
layer intobackbone.body.8.m.0, backbone.body.8.m.1, backbone.body.8.m.2
, but it is otherwise the same exact model.** Note: We are using an older version of nni package
nni==2.4
so I'm not sure if this is resolved already. From the fact that compressor.py still has the assert im inclined to beleive new version will also fail. I'm not an expert on pruning, but I hope this helps :)
Hello @SarBH , I encountered the same problem as you when using nni2.6. Considering the reason of sppf, I decided to use the yolov5 version 5, because the old version still uses the basic spp module, but the pruning process appeared new error, have you succeeded in pruning yolov5?
[2022-01-25 11:21:04] INFO (nni.compression.pytorch.speedup.compressor/MainThread) Update mask for model.12 model.12 node inputs:['4932', 'input.79'] node outputs:['input.107'] file:nni/compression/pytorch/speedup/compressor.py Traceback (most recent call last): File "v12_old_yolo.py", line 85, in
ModelSpeedup(model, dummy_input=dummy_input.to(device), masks_file=masks).speedup_model() File "/nni-master25/nni/compression/pytorch/speedup/compressor.py", line 545, in speedup_model self.infer_modules_masks() File "/nni-master25/nni/compression/pytorch/speedup/compressor.py", line 390, in infer_modules_masks self.update_direct_sparsity(curnode) File "/nni-master25/nni/compression/pytorch/speedup/compressor.py", line 234, in update_direct_sparsity state_dict=copy.deepcopy(module.state_dict()), batch_dim=self.batch_dim) File "/nni-master25/nni/compression/pytorch/speedup/infer_mask.py", line 80, in init self.output = self.module(dummy_input) File "/Users/anaconda3/envs/py37torch17/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(input, **kwargs) TypeError: forward() takes 2 positional arguments but 3 were given
Thanks for the follow up @zhiqwang, and for this awesome project! I investigated where exactly pruning is failing: the compressor.py file asserts that
assert len(node.outputs) == 1, 'The number of the output should be one after the Tuple unpacked manually'
. I see that all other (non failing) nodes in this model indeed have a single output, but this oneMaxPool2d
node has 3 outputs after the update (Below is a printout of the node, see last line):name: backbone.body.8.m, type: module, op_type: MaxPool2d, sub_nodes: ['__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct', '__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_aten::max_pool2d', '__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct', '__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct', '__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct', '__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_aten::max_pool2d', '__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct', '__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct', '__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct', '__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct', '__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_aten::max_pool2d', '__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct', '__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct', '__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct', '__module.backbone/__module.backbone.body/__module.backbone.body.8/__module.backbone.body.8.m_prim::ListConstruct'], inputs: ['input.85'], outputs: ['input.86', 'input.87', '4815'], aux: None
The old hash’s model that doesnt fail prunning actually splits that
backbone.body.8.m
layer intobackbone.body.8.m.0, backbone.body.8.m.1, backbone.body.8.m.2
, but it is otherwise the same exact model. ** Note: We are using an older version of nni packagenni==2.4
so I'm not sure if this is resolved already. From the fact that compressor.py still has the assert im inclined to beleive new version will also fail. I'm not an expert on pruning, but I hope this helps :)Hello @SarBH , I encountered the same problem as you when using nni2.6. Considering the reason of sppf, I decided to use the yolov5 version 5, because the old version still uses the basic spp module, but the pruning process appeared new error, have you succeeded in pruning yolov5?
[2022-01-25 11:21:04] INFO (nni.compression.pytorch.speedup.compressor/MainThread) Update mask for model.12 model.12 node inputs:['4932', 'input.79'] node outputs:['input.107'] file:nni/compression/pytorch/speedup/compressor.py Traceback (most recent call last): File "v12_old_yolo.py", line 85, in ModelSpeedup(model, dummy_input=dummy_input.to(device), masks_file=masks).speedup_model() File "/nni-master25/nni/compression/pytorch/speedup/compressor.py", line 545, in speedup_model self.infer_modules_masks() File "/nni-master25/nni/compression/pytorch/speedup/compressor.py", line 390, in infer_modules_masks self.update_direct_sparsity(curnode) File "/nni-master25/nni/compression/pytorch/speedup/compressor.py", line 234, in update_direct_sparsity state_dict=copy.deepcopy(module.state_dict()), batch_dim=self.batch_dim) File "/nni-master25/nni/compression/pytorch/speedup/infer_mask.py", line 80, in init self.output = self.module(dummy_input) File "/Users/anaconda3/envs/py37torch17/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(input, **kwargs) TypeError: forward() takes 2 positional arguments but 3 were given
Hi, @zhiqwang @syswyl
i met the same error now, have you solved this problem? And can you give me some guidance? Thanks very much!
🐛 Describe the bug
Somewhere between these two commits there was a model backbone change: https://github.com/zhiqwang/yolov5-rt-stack/compare/06022fd...e3e18f2. The three MaxPool2d at
backbone.body.8.m.0, backbone.body.8.m.1, backbone.body.8.m.2
go from being parallel to being serialized in the later hash into justbackbone.body.8.m
with three outputs. (I'm usingnni==2.4
to prune the model, and a node with three outputs is a problem for that)For example, the SMALL model. On the left: commit hash 06022fd, default upstream_version = r4.0 On the right: commit hash e3e18f2, default value for upstream_version changed with the addition of
r6.0
, so I set theupstream_version=r4.0
explicitly. I see the same behavior for'yolov5s', 'yolov5m', 'yolov5l'
All else in the models remained the same, therefore I wondered if this was accidental.
Versions