[Yolact][Pytorch Frontend][Paritioning Issue] Yolact - Instance segmentation model compilation issue

[abdulazizm]

https://github.com/Ma-Dan/yolact/tree/onnx - Ma-Dan version of yolact is onnx exportable (hope its good to JIT trace), tried this with pytorch frontend but getting segmentation fault at partition_for_vitis_ai(mod, params, dpu=target) as below (debug logs enabled - can post full log in file if needed)

DEBUG:pyxir:-- -- Sweep transpose: bX: ['moved_moved_nn_conv2d_NHWC-NCHW-94241211375472'], X: nn.relu-94240959460784, tX: ['nn_conv2d-94241211375472']
DEBUG:pyxir:-- Visit: moved_moved_moved_moved_transpose-94241201609008
-- -- for opt: SweepTransposesFlowDirection
DEBUG:pyxir:-- Visit: nn_bias_add-94241211132064
-- -- for opt: SweepTransposesFlowDirection
DEBUG:pyxir:-- -- Sweep transpose: bX: ['moved_moved_moved_moved_transpose-94241201609008'], X: nn_bias_add-94241211132064, tX: ['nn.relu-94241201608720']
INFO:pyxir:Writing graph visualization to after_partitioning_sweep_transposes.png
Fatal Python error: Segmentation fault

Current thread 0x00007f8944951740 (most recent call first):
  File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow/lib/python3.6/site-packages/tvm-0.8.dev1859+g627e92e7c-py3.6-linux-x86_64.egg/tvm/ir/transform.py", line 161 in __call__
  File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow/lib/python3.6/site-packages/tvm-0.8.dev1859+g627e92e7c-py3.6-linux-x86_64.egg/tvm/relay/op/contrib/vitis_ai.py", line 191 in partition_for_vitis_ai
  File "compile_yoloact_vitisai.py", line 226 in <module>
Segmentation fault (core dumped)

To JIT trace yoloact model:

def do_trace(model, in_size=100):
    model_trace = torch.jit.trace(model, torch.rand(1, 3, in_size, in_size))
    model_trace.eval()
    return model_trace

from yolact import Yolact
from data import cfg, set_cfg, set_dataset
set_cfg('yolact_resnet50' + '_config')
in_size = 550
inp = torch.rand(1, 3, in_size, in_size)

model = Yolact()
model.load_weights("weights/yolact_resnet50_54_800000.pth")
model.eval()
with torch.no_grad():
     out = model(inp)
     script_module = do_trace(model, in_size=550)

Please refer: https://discuss.tvm.apache.org/t/frontend-pytorch-compile-yolact-for-xilinx-hardware-zcu104/11909/5?u=abdulazizm

[abdulazizm]

Hi @jornt-xilinx ,

Please help us with your suggestion for improving the performance. Did you find any irregularities in this mod['main']?

FYI:

1. All nn.conv2d has common 256 output channel but not falls into dpu HW constraint. Anyway to bring those onto dpu?
2. Noticed 1 image.resize2d surrounded by layout transform. Not sure why, I have added it to desired layouts (tried to register with level 10 - but it seems already registered internally), no changed observed in mod['main'] even if I change image.resize2d to 'NCHW' or 'NHWC' layout

fn (%data: Tensor[(1, 3, 550, 550), float32]) -> (Tensor[(1, 19173, 4), float32], Tensor[(1, 19173, 81), float32], Tensor[(1, 19173, 32), float32], Tensor[(19173, 4), float32]) {
  %0 = layout_transform(%data, src_layout="NCHW", dst_layout="NHWC") /* ty=Tensor[(1, 550, 550, 3), float32] */;
  %1 = @tvmgen_default_vitis_ai_main_0(%0) /* ty=(Tensor[(1, 35, 35, 256), float32], Tensor[(1, 69, 69, 256), float32], Tensor[(1, 35, 35, 12), float32], Tensor[(1, 18, 18, 12), float32], Tensor[(1, 9, 9, 12), float32], Tensor[(1, 35, 35, 243), float32], Tensor[(1, 18, 18, 243), float32], Tensor[(1, 9, 9, 243), float32], Tensor[(1, 35, 35, 96), float32], Tensor[(1, 18, 18, 96), float32], Tensor[(1, 9, 9, 96), float32]) */;
  %2 = %1.0;
  %3 = layout_transform(%2, src_layout="NHWC", dst_layout="NCHW") /* ty=Tensor[(1, 256, 35, 35), float32] */;
  %4 = %1.1;
  %5 = image.resize2d(%3, size=[69, 69], roi=[0f, 0f, 0f, 0f], rounding_method="", cubic_alpha=-0.75f) /* C.graph: aten::upsample_bilinear2d, jit._trace.TopLevelTracedModule: __module.fpn */ /* ty=Tensor[(1, 256, 69, 69), float32] */;
  %6 = layout_transform(%4, src_layout="NHWC", dst_layout="NCHW") /* ty=Tensor[(1, 256, 69, 69), float32] */;
  %7 = add(%5, %6) /* C.graph: aten::add, jit._trace.TopLevelTracedModule: __module.fpn */ /* ty=Tensor[(1, 256, 69, 69), float32] */;
  %8 = nn.conv2d(%7, meta[relay.Constant][0] /* ty=Tensor[(256, 256, 3, 3), float32] */, padding=[1, 1, 1, 1], channels=256, kernel_size=[3, 3]) /* C.graph: aten::_convolution, jit._trace.TopLevelTracedModule: __module.fpn/__module.fpn.pred_layers.1_PART_0 */ /* ty=Tensor[(1, 256, 69, 69), float32] */;
  %9 = add(%8, meta[relay.Constant][1] /* ty=Tensor[(1, 256, 1, 1), float32] */) /* ty=Tensor[(1, 256, 69, 69), float32] */;
  %10 = nn.relu(%9) /* C.graph: aten::relu, jit._trace.TopLevelTracedModule: __module.fpn */ /* ty=Tensor[(1, 256, 69, 69), float32] */;
  %11 = nn.conv2d(%10, meta[relay.Constant][2] /* ty=Tensor[(256, 256, 3, 3), float32] */, padding=[1, 1, 1, 1], channels=256, kernel_size=[3, 3]) /* C.graph: aten::_convolution, jit._trace.TopLevelTracedModule: __module.prediction_layers.0/__module.prediction_layers.0.upfeature/__module.prediction_layers.0.upfeature.0_PART_2 */ /* ty=Tensor[(1, 256, 69, 69), float32] */;
  %12 = add(%11, meta[relay.Constant][3] /* ty=Tensor[(1, 256, 1, 1), float32] */) /* ty=Tensor[(1, 256, 69, 69), float32] */;
  %13 = nn.relu(%12) /* C.graph: aten::relu_, jit._trace.TopLevelTracedModule: __module.prediction_layers.0/__module.prediction_layers.0.upfeature/__module.prediction_layers.0.upfeature.1 */ /* ty=Tensor[(1, 256, 69, 69), float32] */;
  %14 = nn.conv2d(%13, meta[relay.Constant][4] /* ty=Tensor[(12, 256, 3, 3), float32] */, padding=[1, 1, 1, 1], channels=12, kernel_size=[3, 3]) /* C.graph: aten::_convolution, jit._trace.TopLevelTracedModule: __module.prediction_layers.0/__module.prediction_layers.0.bbox_layer_PART_0 */ /* ty=Tensor[(1, 12, 69, 69), float32] */;
  %15 = add(%14, meta[relay.Constant][5] /* ty=Tensor[(1, 12, 1, 1), float32] */) /* ty=Tensor[(1, 12, 69, 69), float32] */;
  %16 = transpose(%15, axes=[0, 2, 3, 1]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.0 */ /* ty=Tensor[(1, 69, 69, 12), float32] */;
  %17 = %1.2;
  %18 = transpose(%17, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.1 */ /* ty=Tensor[(1, 35, 35, 12), float32] */;
  %19 = %1.3;
  %20 = transpose(%19, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.2 */ /* ty=Tensor[(1, 18, 18, 12), float32] */;
  %21 = %1.4;
  %22 = transpose(%21, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.3 */ /* ty=Tensor[(1, 9, 9, 12), float32] */;
  %23 = reshape(%16, newshape=[1, -1, 4]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.0 */ /* ty=Tensor[(1, 14283, 4), float32] */;
  %24 = reshape(%18, newshape=[1, -1, 4]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.1 */ /* ty=Tensor[(1, 3675, 4), float32] */;
  %25 = reshape(%20, newshape=[1, -1, 4]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.2 */ /* ty=Tensor[(1, 972, 4), float32] */;
  %26 = reshape(%22, newshape=[1, -1, 4]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.3 */ /* ty=Tensor[(1, 243, 4), float32] */;
  %27 = (%23, %24, %25, %26);
  %28 = nn.conv2d(%13, meta[relay.Constant][6] /* ty=Tensor[(243, 256, 3, 3), float32] */, padding=[1, 1, 1, 1], channels=243, kernel_size=[3, 3]) /* C.graph: aten::_convolution, jit._trace.TopLevelTracedModule: __module.prediction_layers.0/__module.prediction_layers.0.conf_layer_PART_0 */ /* ty=Tensor[(1, 243, 69, 69), float32] */;
  %29 = add(%28, meta[relay.Constant][7] /* ty=Tensor[(1, 243, 1, 1), float32] */) /* ty=Tensor[(1, 243, 69, 69), float32] */;
  %30 = transpose(%29, axes=[0, 2, 3, 1]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.0 */ /* ty=Tensor[(1, 69, 69, 243), float32] */;
  %31 = %1.5;
  %32 = transpose(%31, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.1 */ /* ty=Tensor[(1, 35, 35, 243), float32] */;
  %33 = %1.6;
  %34 = transpose(%33, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.2 */ /* ty=Tensor[(1, 18, 18, 243), float32] */;
  %35 = %1.7;
  %36 = transpose(%35, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.3 */ /* ty=Tensor[(1, 9, 9, 243), float32] */;
  %37 = reshape(%30, newshape=[1, -1, 81]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.0 */ /* ty=Tensor[(1, 14283, 81), float32] */;
  %38 = reshape(%32, newshape=[1, -1, 81]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.1 */ /* ty=Tensor[(1, 3675, 81), float32] */;
  %39 = reshape(%34, newshape=[1, -1, 81]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.2 */ /* ty=Tensor[(1, 972, 81), float32] */;
  %40 = reshape(%36, newshape=[1, -1, 81]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.3 */ /* ty=Tensor[(1, 243, 81), float32] */;
  %41 = (%37, %38, %39, %40);
  %42 = concatenate(%41, axis=-2) /* C.graph: aten::cat, warning: no trace info 1 */ /* ty=Tensor[(1, 19173, 81), float32] */;
  %43 = nn.conv2d(%13, meta[relay.Constant][8] /* ty=Tensor[(96, 256, 3, 3), float32] */, padding=[1, 1, 1, 1], channels=96, kernel_size=[3, 3]) /* C.graph: aten::_convolution, jit._trace.TopLevelTracedModule: __module.prediction_layers.0/__module.prediction_layers.0.mask_layer_PART_0 */ /* ty=Tensor[(1, 96, 69, 69), float32] */;
  %44 = add(%43, meta[relay.Constant][9] /* ty=Tensor[(1, 96, 1, 1), float32] */) /* ty=Tensor[(1, 96, 69, 69), float32] */;
  %45 = transpose(%44, axes=[0, 2, 3, 1]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.0 */ /* ty=Tensor[(1, 69, 69, 96), float32] */;
  %46 = reshape(%45, newshape=[1, -1, 32]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.0 */ /* ty=Tensor[(1, 14283, 32), float32] */;
  %47 = %1.8;
  %48 = transpose(%47, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.1 */ /* ty=Tensor[(1, 35, 35, 96), float32] */;
  %49 = reshape(%48, newshape=[1, -1, 32]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.1 */ /* ty=Tensor[(1, 3675, 32), float32] */;
  %50 = %1.9;
  %51 = transpose(%50, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.2 */ /* ty=Tensor[(1, 18, 18, 96), float32] */;
  %52 = reshape(%51, newshape=[1, -1, 32]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.2 */ /* ty=Tensor[(1, 972, 32), float32] */;
  %53 = %1.10;
  %54 = transpose(%53, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.3 */ /* ty=Tensor[(1, 9, 9, 96), float32] */;
  %55 = reshape(%54, newshape=[1, -1, 32]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.3 */ /* ty=Tensor[(1, 243, 32), float32] */;
  %56 = tanh(%46) /* C.graph: aten::tanh, jit._trace.TopLevelTracedModule: __module.prediction_layers.0 */ /* ty=Tensor[(1, 14283, 32), float32] */;
  %57 = tanh(%49) /* C.graph: aten::tanh, jit._trace.TopLevelTracedModule: __module.prediction_layers.1 */ /* ty=Tensor[(1, 3675, 32), float32] */;
  %58 = tanh(%52) /* C.graph: aten::tanh, jit._trace.TopLevelTracedModule: __module.prediction_layers.2 */ /* ty=Tensor[(1, 972, 32), float32] */;
  %59 = tanh(%55) /* C.graph: aten::tanh, jit._trace.TopLevelTracedModule: __module.prediction_layers.3 */ /* ty=Tensor[(1, 243, 32), float32] */;
  %60 = (%56, %57, %58, %59);
  %61 = concatenate(%27, axis=-2) /* C.graph: aten::cat, warning: no trace info 0 */ /* ty=Tensor[(1, 19173, 4), float32] */;
  %62 = nn.softmax(%42) /* C.graph: aten::softmax, warning: no trace info 4 */ /* ty=Tensor[(1, 19173, 81), float32] */;
  %63 = concatenate(%60, axis=-2) /* C.graph: aten::cat, warning: no trace info 2 */ /* ty=Tensor[(1, 19173, 32), float32] */;
  (%61, %62, %63, meta[relay.Constant][10] /* ty=Tensor[(19173, 4), float32] */)
}

[jtuyls]

@abdulazizm I think the issue is the bilinear resize2d layers. From the Vitis AI user guide:

If the mode of the resize is 'BILINEAR', align_corner=false, half_pixel_centers = false, size = 2, 4, 8; align_corner=false, half_pixel_centers = true, size = 2, 4 can be transformed to DPU implementations (pad +depthwisetransposed conv2d). Ifthe mode of the resize is 'NEAREST' and the size is an integer, the resize would be mapped to DPU implementations.

I created a small test case and the DPU compiler couldn't handle these resize layers. I believe the issue is the floating scale from [35, 35] to [69, 69].

[abdulazizm]

Thanks for the quick reply, @jornt-xilinx . Completely makes sense. Are there any suggestions to improve this further by any chance?

[abdulazizm]

@jtuyls aten::view, aten::permute operators are running on cpu it seems. DPU is not supporting these?

Tried changing aten::view -> aten::reshape , seems its also pulled on CPU.

Is this expected?

[abdulazizm]

@abdulazizm I think the issue is the bilinear resize2d layers. From the Vitis AI user guide:

If the mode of the resize is 'BILINEAR', align_corner=false, half_pixel_centers = false, size = 2, 4, 8; align_corner=false, half_pixel_centers = true, size = 2, 4 can be transformed to DPU implementations (pad +depthwisetransposed conv2d). Ifthe mode of the resize is 'NEAREST' and the size is an integer, the resize would be mapped to DPU implementations.

I created a small test case and the DPU compiler couldn't handle these resize layers. I believe the issue is the floating scale from [35, 35] to [69, 69].

@jtuyls I don't think the issue is with scaling_factor. Refer below mod['main'] -> this has resize2d [69,69] to [138,138] in CPU after partition.

fn (%data: Tensor[(1, 3, 550, 550), float32]) -> (Tensor[(1, 19173, 4), float32], Tensor[(1, 19173, 81), float32], Tensor[(1, 19173, 32), float32], Tensor[(19173, 4), float32], Tensor[(1, 138, 138, 32), float32]) {
  %0 = layout_transform(%data, src_layout="NCHW", dst_layout="NHWC") /* ty=Tensor[(1, 550, 550, 3), float32] */;
  %1 = @tvmgen_default_vitis_ai_main_0(%0) /* ty=(Tensor[(1, 69, 69, 12), float32], Tensor[(1, 35, 35, 12), float32], Tensor[(1, 18, 18, 12), float32], Tensor[(1, 9, 9, 12), float32], Tensor[(1, 69, 69, 243), float32], Tensor[(1, 35, 35, 243), float32], Tensor[(1, 18, 18, 243), float32], Tensor[(1, 9, 9, 243), float32], Tensor[(1, 69, 69, 96), float32], Tensor[(1, 35, 35, 96), float32], Tensor[(1, 18, 18, 96), float32], Tensor[(1, 9, 9, 96), float32], Tensor[(1, 69, 69, 256), float32]) */;
  %2 = %1.0;
  %3 = transpose(%2, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.0 */ /* ty=Tensor[(1, 69, 69, 12), float32] */;
  %4 = %1.1;
  %5 = transpose(%4, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.1 */ /* ty=Tensor[(1, 35, 35, 12), float32] */;
  %6 = %1.2;
  %7 = transpose(%6, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.2 */ /* ty=Tensor[(1, 18, 18, 12), float32] */;
  %8 = %1.3;
  %9 = transpose(%8, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.3 */ /* ty=Tensor[(1, 9, 9, 12), float32] */;
  %10 = reshape(%3, newshape=[1, -1, 4]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.0 */ /* ty=Tensor[(1, 14283, 4), float32] */;
  %11 = reshape(%5, newshape=[1, -1, 4]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.1 */ /* ty=Tensor[(1, 3675, 4), float32] */;
  %12 = reshape(%7, newshape=[1, -1, 4]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.2 */ /* ty=Tensor[(1, 972, 4), float32] */;
  %13 = reshape(%9, newshape=[1, -1, 4]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.3 */ /* ty=Tensor[(1, 243, 4), float32] */;
  %14 = (%10, %11, %12, %13);
  %15 = %1.4;
  %16 = transpose(%15, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.0 */ /* ty=Tensor[(1, 69, 69, 243), float32] */;
  %17 = %1.5;
  %18 = transpose(%17, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.1 */ /* ty=Tensor[(1, 35, 35, 243), float32] */;
  %19 = %1.6;
  %20 = transpose(%19, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.2 */ /* ty=Tensor[(1, 18, 18, 243), float32] */;
  %21 = %1.7;
  %22 = transpose(%21, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.3 */ /* ty=Tensor[(1, 9, 9, 243), float32] */;
  %23 = reshape(%16, newshape=[1, -1, 81]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.0 */ /* ty=Tensor[(1, 14283, 81), float32] */;
  %24 = reshape(%18, newshape=[1, -1, 81]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.1 */ /* ty=Tensor[(1, 3675, 81), float32] */;
  %25 = reshape(%20, newshape=[1, -1, 81]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.2 */ /* ty=Tensor[(1, 972, 81), float32] */;
  %26 = reshape(%22, newshape=[1, -1, 81]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.3 */ /* ty=Tensor[(1, 243, 81), float32] */;
  %27 = (%23, %24, %25, %26);
  %28 = concatenate(%27, axis=-2) /* C.graph: aten::cat, warning: no trace info 4 */ /* ty=Tensor[(1, 19173, 81), float32] */;
  %29 = %1.8;
  %30 = transpose(%29, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.0 */ /* ty=Tensor[(1, 69, 69, 96), float32] */;
  %31 = reshape(%30, newshape=[1, -1, 32]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.0 */ /* ty=Tensor[(1, 14283, 32), float32] */;
  %32 = %1.9;
  %33 = transpose(%32, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.1 */ /* ty=Tensor[(1, 35, 35, 96), float32] */;
  %34 = reshape(%33, newshape=[1, -1, 32]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.1 */ /* ty=Tensor[(1, 3675, 32), float32] */;
  %35 = %1.10;
  %36 = transpose(%35, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.2 */ /* ty=Tensor[(1, 18, 18, 96), float32] */;
  %37 = reshape(%36, newshape=[1, -1, 32]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.2 */ /* ty=Tensor[(1, 972, 32), float32] */;
  %38 = %1.11;
  %39 = transpose(%38, axes=[0, 1, 2, 3]) /* C.graph: aten::permute, jit._trace.TopLevelTracedModule: __module.prediction_layers.3 */ /* ty=Tensor[(1, 9, 9, 96), float32] */;
  %40 = reshape(%39, newshape=[1, -1, 32]) /* C.graph: aten::view, jit._trace.TopLevelTracedModule: __module.prediction_layers.3 */ /* ty=Tensor[(1, 243, 32), float32] */;
  %41 = tanh(%31) /* C.graph: aten::tanh, jit._trace.TopLevelTracedModule: __module.prediction_layers.0 */ /* ty=Tensor[(1, 14283, 32), float32] */;
  %42 = tanh(%34) /* C.graph: aten::tanh, jit._trace.TopLevelTracedModule: __module.prediction_layers.1 */ /* ty=Tensor[(1, 3675, 32), float32] */;
  %43 = tanh(%37) /* C.graph: aten::tanh, jit._trace.TopLevelTracedModule: __module.prediction_layers.2 */ /* ty=Tensor[(1, 972, 32), float32] */;
  %44 = tanh(%40) /* C.graph: aten::tanh, jit._trace.TopLevelTracedModule: __module.prediction_layers.3 */ /* ty=Tensor[(1, 243, 32), float32] */;
  %45 = (%41, %42, %43, %44);
  %46 = %1.12;
  %47 = layout_transform(%46, src_layout="NHWC", dst_layout="NCHW") /* ty=Tensor[(1, 256, 69, 69), float32] */;
  %48 = image.resize2d(%47, size=[138, 138], roi=[0f, 0f, 0f, 0f], rounding_method="", cubic_alpha=-0.75f) /* C.graph: aten::upsample_bilinear2d, jit._trace.TopLevelTracedModule: __module.proto_net/__module.proto_net.6 */ /* ty=Tensor[(1, 256, 138, 138), float32] */;
  %49 = nn.relu(%48) /* C.graph: aten::relu_, jit._trace.TopLevelTracedModule: __module.proto_net/__module.proto_net.7 */ /* ty=Tensor[(1, 256, 138, 138), float32] */;
  %50 = nn.conv2d(%49, meta[relay.Constant][1] /* ty=Tensor[(32, 256, 1, 1), float32] */, padding=[0, 0, 0, 0], channels=32, kernel_size=[1, 1]) /* C.graph: aten::_convolution, jit._trace.TopLevelTracedModule: __module.proto_net/__module.proto_net.8_PART_0 */ /* ty=Tensor[(1, 32, 138, 138), float32] */;
  %51 = add(%50, meta[relay.Constant][2] /* ty=Tensor[(1, 32, 1, 1), float32] */) /* ty=Tensor[(1, 32, 138, 138), float32] */;
  %52 = nn.relu(%51) /* C.graph: aten::relu_, warning: no trace info 0 */ /* ty=Tensor[(1, 32, 138, 138), float32] */;
  %53 = concatenate(%14, axis=-2) /* C.graph: aten::cat, warning: no trace info 3 */ /* ty=Tensor[(1, 19173, 4), float32] */;
  %54 = nn.softmax(%28) /* C.graph: aten::softmax, warning: no trace info 7 */ /* ty=Tensor[(1, 19173, 81), float32] */;
  %55 = concatenate(%45, axis=-2) /* C.graph: aten::cat, warning: no trace info 5 */ /* ty=Tensor[(1, 19173, 32), float32] */;
  %56 = transpose(%52, axes=[0, 2, 3, 1]) /* C.graph: aten::permute, warning: no trace info 1 */ /* ty=Tensor[(1, 138, 138, 32), float32] */;
  (%53, %54, %55, meta[relay.Constant][0] /* ty=Tensor[(19173, 4), float32] */, %56)
}

Is there any timeline for supporting bilinear interpolation with this tvm flow?

[jtuyls]

@abdulazizm I think that this (69, 69) -> (128, 128) resize could be offloaded to the DPU indeed, however, the tanh and reshape ops in between are prohibiting putting resize2d in the DPU subgraph (we only support one DPU subgraph).