search

Xilinx / Vitis-AI

Vitis AI is Xilinx’s development stack for AI inference on Xilinx hardware platforms, including both edge devices and Alveo cards.
https://www.xilinx.com/ai
Apache License 2.0
1.49k stars 634 forks source link

KeyError in vai_q_pytorch at inspect phase when the model contain torch.nn.functional.Conv2d #986

Open KeepHoper opened 2 years ago

KeepHoper commented 2 years ago

Hello,

I'm trying to quantize a PyTorch model with Vitis-AI 2.5 follow the user guide(UG 1414). When the model contains a torch.nn.functional.conv2d operation, and I actually need to use one output of the network as a convolution kernel to convolve another output, I get a KeyError in the Processing ops, the complete log is as follows:

[VAIQ_NOTE]: Loading NNDCT kernels...

[VAIQ_NOTE]: =>Inspector is initialized successfully with target: name: DPUCAHX8L_ISA0_SP type: DPUCAHX8L isa_version: 0

[VAIQ_NOTE]: =>Start to inspect model...

[VAIQ_NOTE]: =>Quant Module is in 'cpu'.

[VAIQ_NOTE]: =>Parsing SiameseFC...

[VAIQ_NOTE]: Start to trace model... /workspace/siamfc/model.py:93: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! match_map = fct.conv2d(embedding_search.view(1, b * c, h, w), embedding_template, stride=1, groups=b)

[VAIQ_NOTE]: Finish tracing.

[VAIQ_NOTE]: Processing ops... ██████████████████████████████████████████████████| 49/49 [00:00<00:00, 3035.49i Traceback (most recent call last): File "test_quant.py", line 34, in inspector.inspect(model, (x1, x2), device=device, output_dir="inspect", image_format="png") File "/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/site-packages/pytorch_nndct/apis.py", line 182, in inspect self._inspector_impl.inspect(module, input_args, device, output_dir, verbose_level) File "/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/site-packages/pytorch_nndct/hardware/inspector.py", line 83, in inspect dev_graph = self._prepare_deployable_graph(copied_model, input_args, device, output_dir) File "/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/site-packages/pytorch_nndct/hardware/inspector.py", line 96, in _prepare_deployable_graph device=device) File "/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/site-packages/pytorch_nndct/qproc/utils.py", line 193, in prepare_quantizable_module graph = parse_module(module, input_args) File "/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/site-packages/pytorch_nndct/qproc/utils.py", line 83, in parse_module module, input_args) File "/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/site-packages/pytorch_nndct/parse/parser.py", line 77, in call self._load_data(nndct_graph, module) File "/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/site-packages/pytorch_nndct/parse/parser.py", line 346, in _load_data data = module.state_dict()[get_short_name(tensor.name)].cpu().numpy() KeyError: '2218'

The definition of the model is as follows:

class SiameseFC(nn.Module):
    def __init__(self, embedding_net, upscale=False, upscale_size=None):
        super(SiameseFC, self).__init__()
        self.embedding_net = resnet18()
        self.match_BatchNorm = nn.BatchNorm2d(1)

    def forward(self, x1, x2):
        embedding_template = self.embedding_net(x1)
        embedding_search = self.embedding_net(x2)

        b, c, h, w = embedding_search.shape

        match_map = fct.conv2d(embedding_search.view(1, b * c, h, w), embedding_template, stride=1, groups=b)

        match_map = match_map.permute(1, 0, 2, 3)
        match_map = self.match_BatchNorm(match_map)

        return match_map

Does this mean that model checking does not support torch.nn.functional.conv2d operation, is there any solution, or if this issue is ignored, will problems be encountered in the model quantification process?

Thanks!

carrotandjzy commented 2 years ago

In vai-2.5, we don't support torch.nn.functional.conv2d in quantizer yet. Can you replace this op with nn.Conv2d ?

KeepHoper commented 2 years ago

In vai-2.5, we don't support torch.nn.functional.conv2d in quantizer yet. Can you replace this op with nn.Conv2d ? I actually need to use the convolution kernel with custom weights, so nn.Conv2d won't work.

MingyangZhang77 commented 1 year ago

Hi. Is torch.nn.functional.conv2d supported in vai-3.0? Or is there any workaround? I am facing this problem with my project, too.

KeepHoper commented 1 year ago

Hi. Is torch.nn.functional.conv2d supported in vai-3.0? Or is there any workaround? I am facing this problem with my project, too.

As far as I know, vitis ai 3.0 does not yet support torch.nn.functional.conv2d. Maybe you can consider using matrix multiplication to implement your operation in your project.

Good luck！

MingyangZhang77 commented 1 year ago

Hi. Is torch.nn.functional.conv2d supported in vai-3.0? Or is there any workaround? I am facing this problem with my project, too.

As far as I know, vitis ai 3.0 does not yet support torch.nn.functional.conv2d. Maybe you can consider using matrix multiplication to implement your operation in your project.

Good luck！

Thanks for your advice! I will try it!