No optimization profile defined

[EulersBoiler]

I would like to load an .onnx of a segmentation network i trained to run on a Jetson Nano 2GB. When loading the network with jetson.inference.net() the engine is started to build, but it fails due to a missing optimization profile. I saw a tag --profile to pass a profile as an argument, but i do not see any references to how i create and save an optimization profile.

I saw the tutorial on dynamic shapes https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#work_dynamic_shapes, but not how to save and load such a profile

Thanks for your help

[dusty-nv]

Hi @EulersBoiler, what segmentation model are you using? jetson.inference.segNet is setup to use FCN-ResNet and FCN-Alexnet. FCN-Alexnet is legacy model from caffe (no longer used for new models really), whereas FCN-ResNet is exported from PyTorch to ONNX with fixed shapes, so it doesn't need the dynamic shape profiles.

For a tutorial on training FCN-ResNet with PyTorch, see here: https://www.highvoltagecode.com/post/edge-ai-semantic-segmentation-on-nvidia-jetson

[EulersBoiler]

Hi Dusty, I am using a custom model based on a resnet18

[EulersBoiler]

I further looked into my model and cant seem to find any dynamic parts. It all seems to have known values. Down below the graph printed by the onnx module for python

graph Network (
  %data[FLOAT, BatchSizex3x224x224]
) initializers (
  %data_Mean[FLOAT, 1x3x1x1]
  %data_StandardDeviation[FLOAT, 1x3x1x1]
  %conv1_W[FLOAT, 64x3x7x7]
  %conv1_B[FLOAT, 64]
  %bn_conv1_scale[FLOAT, 64]
  %bn_conv1_B[FLOAT, 64]
  %bn_conv1_mean[FLOAT, 64]
  %bn_conv1_var[FLOAT, 64]
  %res2a_branch2a_W[FLOAT, 64x64x3x3]
  %res2a_branch2a_B[FLOAT, 64]
  %bn2a_branch2a_scale[FLOAT, 64]
  %bn2a_branch2a_B[FLOAT, 64]
  %bn2a_branch2a_mean[FLOAT, 64]
  %bn2a_branch2a_var[FLOAT, 64]
  %res2a_branch2b_W[FLOAT, 64x64x3x3]
  %res2a_branch2b_B[FLOAT, 64]
  %bn2a_branch2b_scale[FLOAT, 64]
  %bn2a_branch2b_B[FLOAT, 64]
  %bn2a_branch2b_mean[FLOAT, 64]
  %bn2a_branch2b_var[FLOAT, 64]
  %res2b_branch2a_W[FLOAT, 64x64x3x3]
  %res2b_branch2a_B[FLOAT, 64]
  %bn2b_branch2a_scale[FLOAT, 64]
  %bn2b_branch2a_B[FLOAT, 64]
  %bn2b_branch2a_mean[FLOAT, 64]
  %bn2b_branch2a_var[FLOAT, 64]
  %res2b_branch2b_W[FLOAT, 64x64x3x3]
  %res2b_branch2b_B[FLOAT, 64]
  %bn2b_branch2b_scale[FLOAT, 64]
  %bn2b_branch2b_B[FLOAT, 64]
  %bn2b_branch2b_mean[FLOAT, 64]
  %bn2b_branch2b_var[FLOAT, 64]
  %res3a_branch2a_W[FLOAT, 128x64x3x3]
  %res3a_branch2a_B[FLOAT, 128]
  %bn3a_branch2a_scale[FLOAT, 128]
  %bn3a_branch2a_B[FLOAT, 128]
  %bn3a_branch2a_mean[FLOAT, 128]
  %bn3a_branch2a_var[FLOAT, 128]
  %res3a_branch2b_W[FLOAT, 128x128x3x3]
  %res3a_branch2b_B[FLOAT, 128]
  %bn3a_branch2b_scale[FLOAT, 128]
  %bn3a_branch2b_B[FLOAT, 128]
  %bn3a_branch2b_mean[FLOAT, 128]
  %bn3a_branch2b_var[FLOAT, 128]
  %res3a_branch1_W[FLOAT, 128x64x1x1]
  %res3a_branch1_B[FLOAT, 128]
  %bn3a_branch1_scale[FLOAT, 128]
  %bn3a_branch1_B[FLOAT, 128]
  %bn3a_branch1_mean[FLOAT, 128]
  %bn3a_branch1_var[FLOAT, 128]
  %res3b_branch2a_W[FLOAT, 128x128x3x3]
  %res3b_branch2a_B[FLOAT, 128]
  %bn3b_branch2a_scale[FLOAT, 128]
  %bn3b_branch2a_B[FLOAT, 128]
  %bn3b_branch2a_mean[FLOAT, 128]
  %bn3b_branch2a_var[FLOAT, 128]
  %res3b_branch2b_W[FLOAT, 128x128x3x3]
  %res3b_branch2b_B[FLOAT, 128]
  %bn3b_branch2b_scale[FLOAT, 128]
  %bn3b_branch2b_B[FLOAT, 128]
  %bn3b_branch2b_mean[FLOAT, 128]
  %bn3b_branch2b_var[FLOAT, 128]
  %res4a_branch2a_W[FLOAT, 256x128x3x3]
  %res4a_branch2a_B[FLOAT, 256]
  %bn4a_branch2a_scale[FLOAT, 256]
  %bn4a_branch2a_B[FLOAT, 256]
  %bn4a_branch2a_mean[FLOAT, 256]
  %bn4a_branch2a_var[FLOAT, 256]
  %res4a_branch2b_W[FLOAT, 256x256x3x3]
  %res4a_branch2b_B[FLOAT, 256]
  %bn4a_branch2b_scale[FLOAT, 256]
  %bn4a_branch2b_B[FLOAT, 256]
  %bn4a_branch2b_mean[FLOAT, 256]
  %bn4a_branch2b_var[FLOAT, 256]
  %res4a_branch1_W[FLOAT, 256x128x1x1]
  %res4a_branch1_B[FLOAT, 256]
  %bn4a_branch1_scale[FLOAT, 256]
  %bn4a_branch1_B[FLOAT, 256]
  %bn4a_branch1_mean[FLOAT, 256]
  %bn4a_branch1_var[FLOAT, 256]
  %res4b_branch2a_W[FLOAT, 256x256x3x3]
  %res4b_branch2a_B[FLOAT, 256]
  %bn4b_branch2a_scale[FLOAT, 256]
  %bn4b_branch2a_B[FLOAT, 256]
  %bn4b_branch2a_mean[FLOAT, 256]
  %bn4b_branch2a_var[FLOAT, 256]
  %res4b_branch2b_W[FLOAT, 256x256x3x3]
  %res4b_branch2b_B[FLOAT, 256]
  %bn4b_branch2b_scale[FLOAT, 256]
  %bn4b_branch2b_B[FLOAT, 256]
  %bn4b_branch2b_mean[FLOAT, 256]
  %bn4b_branch2b_var[FLOAT, 256]
  %res5a_branch2a_W[FLOAT, 512x256x3x3]
  %res5a_branch2a_B[FLOAT, 512]
  %bn5a_branch2a_scale[FLOAT, 512]
  %bn5a_branch2a_B[FLOAT, 512]
  %bn5a_branch2a_mean[FLOAT, 512]
  %bn5a_branch2a_var[FLOAT, 512]
  %res5a_branch2b_W[FLOAT, 512x512x3x3]
  %res5a_branch2b_B[FLOAT, 512]
  %bn5a_branch2b_scale[FLOAT, 512]
  %bn5a_branch2b_B[FLOAT, 512]
  %bn5a_branch2b_mean[FLOAT, 512]
  %bn5a_branch2b_var[FLOAT, 512]
  %res5a_branch1_W[FLOAT, 512x256x1x1]
  %res5a_branch1_B[FLOAT, 512]
  %bn5a_branch1_scale[FLOAT, 512]
  %bn5a_branch1_B[FLOAT, 512]
  %bn5a_branch1_mean[FLOAT, 512]
  %bn5a_branch1_var[FLOAT, 512]
  %res5b_branch2a_W[FLOAT, 512x512x3x3]
  %res5b_branch2a_B[FLOAT, 512]
  %bn5b_branch2a_scale[FLOAT, 512]
  %bn5b_branch2a_B[FLOAT, 512]
  %bn5b_branch2a_mean[FLOAT, 512]
  %bn5b_branch2a_var[FLOAT, 512]
  %res5b_branch2b_W[FLOAT, 512x512x3x3]
  %res5b_branch2b_B[FLOAT, 512]
  %bn5b_branch2b_scale[FLOAT, 512]
  %bn5b_branch2b_B[FLOAT, 512]
  %bn5b_branch2b_mean[FLOAT, 512]
  %bn5b_branch2b_var[FLOAT, 512]
  %aspp_Conv_1_W[FLOAT, 256x512x1x1]
  %aspp_Conv_1_B[FLOAT, 256]
  %aspp_BatchNorm_1_scale[FLOAT, 256]
  %aspp_BatchNorm_1_B[FLOAT, 256]
  %aspp_BatchNorm_1_mean[FLOAT, 256]
  %aspp_BatchNorm_1_var[FLOAT, 256]
  %aspp_Conv_2_W[FLOAT, 256x512x3x3]
  %aspp_Conv_2_B[FLOAT, 256]
  %aspp_BatchNorm_2_scale[FLOAT, 256]
  %aspp_BatchNorm_2_B[FLOAT, 256]
  %aspp_BatchNorm_2_mean[FLOAT, 256]
  %aspp_BatchNorm_2_var[FLOAT, 256]
  %aspp_Conv_3_W[FLOAT, 256x512x3x3]
  %aspp_Conv_3_B[FLOAT, 256]
  %aspp_BatchNorm_3_scale[FLOAT, 256]
  %aspp_BatchNorm_3_B[FLOAT, 256]
  %aspp_BatchNorm_3_mean[FLOAT, 256]
  %aspp_BatchNorm_3_var[FLOAT, 256]
  %aspp_Conv_4_W[FLOAT, 256x512x3x3]
  %aspp_Conv_4_B[FLOAT, 256]
  %aspp_BatchNorm_4_scale[FLOAT, 256]
  %aspp_BatchNorm_4_B[FLOAT, 256]
  %aspp_BatchNorm_4_mean[FLOAT, 256]
  %aspp_BatchNorm_4_var[FLOAT, 256]
  %dec_c1_W[FLOAT, 256x1024x1x1]
  %dec_c1_B[FLOAT, 256]
  %dec_bn1_scale[FLOAT, 256]
  %dec_bn1_B[FLOAT, 256]
  %dec_bn1_mean[FLOAT, 256]
  %dec_bn1_var[FLOAT, 256]
  %dec_upsample1_W[FLOAT, 256x256x8x8]
  %dec_upsample1_B[FLOAT, 256]
  %dec_c2_W[FLOAT, 48x64x1x1]
  %dec_c2_B[FLOAT, 48]
  %dec_bn2_scale[FLOAT, 48]
  %dec_bn2_B[FLOAT, 48]
  %dec_bn2_mean[FLOAT, 48]
  %dec_bn2_var[FLOAT, 48]
  %dec_c3_W[FLOAT, 256x304x3x3]
  %dec_c3_B[FLOAT, 256]
  %dec_bn3_scale[FLOAT, 256]
  %dec_bn3_B[FLOAT, 256]
  %dec_bn3_mean[FLOAT, 256]
  %dec_bn3_var[FLOAT, 256]
  %dec_c4_W[FLOAT, 256x256x3x3]
  %dec_c4_B[FLOAT, 256]
  %dec_bn4_scale[FLOAT, 256]
  %dec_bn4_B[FLOAT, 256]
  %dec_bn4_mean[FLOAT, 256]
  %dec_bn4_var[FLOAT, 256]
  %scorer_W[FLOAT, 2x256x1x1]
  %scorer_B[FLOAT, 2]
  %dec_upsample2_W[FLOAT, 2x2x8x8]
  %dec_upsample2_B[FLOAT, 2]
) {
  %data_Sub = Sub(%data, %data_Mean)
  %data_Div = Div(%data_Sub, %data_StandardDeviation)
  %conv1 = Conv[dilations = [1, 1], group = 1, kernel_shape = [7, 7], pads = [3, 3, 3, 3], strides = [2, 2]](%data_Div, %conv1_W, %conv1_B)
  %bn_conv1 = BatchNormalization[epsilon = 9.99999974737875e-06](%conv1, %bn_conv1_scale, %bn_conv1_B, %bn_conv1_mean, %bn_conv1_var)
  %conv1_relu = Relu(%bn_conv1)
  %pool1 = MaxPool[kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [2, 2]](%conv1_relu)
  %res2a_branch2a = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%pool1, %res2a_branch2a_W, %res2a_branch2a_B)
  %bn2a_branch2a = BatchNormalization[epsilon = 9.99999974737875e-06](%res2a_branch2a, %bn2a_branch2a_scale, %bn2a_branch2a_B, %bn2a_branch2a_mean, %bn2a_branch2a_var)
  %res2a_branch2a_relu = Relu(%bn2a_branch2a)
  %res2a_branch2b = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%res2a_branch2a_relu, %res2a_branch2b_W, %res2a_branch2b_B)
  %bn2a_branch2b = BatchNormalization[epsilon = 9.99999974737875e-06](%res2a_branch2b, %bn2a_branch2b_scale, %bn2a_branch2b_B, %bn2a_branch2b_mean, %bn2a_branch2b_var)
  %res2a = Sum(%bn2a_branch2b, %pool1)
  %res2a_relu = Relu(%res2a)
  %res2b_branch2a = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%res2a_relu, %res2b_branch2a_W, %res2b_branch2a_B)
  %bn2b_branch2a = BatchNormalization[epsilon = 9.99999974737875e-06](%res2b_branch2a, %bn2b_branch2a_scale, %bn2b_branch2a_B, %bn2b_branch2a_mean, %bn2b_branch2a_var)
  %res2b_branch2a_relu = Relu(%bn2b_branch2a)
  %res2b_branch2b = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%res2b_branch2a_relu, %res2b_branch2b_W, %res2b_branch2b_B)
  %bn2b_branch2b = BatchNormalization[epsilon = 9.99999974737875e-06](%res2b_branch2b, %bn2b_branch2b_scale, %bn2b_branch2b_B, %bn2b_branch2b_mean, %bn2b_branch2b_var)
  %res2b = Sum(%bn2b_branch2b, %res2a_relu)
  %res2b_relu = Relu(%res2b)
  %res3a_branch2a = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [2, 2]](%res2b_relu, %res3a_branch2a_W, %res3a_branch2a_B)
  %bn3a_branch2a = BatchNormalization[epsilon = 9.99999974737875e-06](%res3a_branch2a, %bn3a_branch2a_scale, %bn3a_branch2a_B, %bn3a_branch2a_mean, %bn3a_branch2a_var)
  %res3a_branch2a_relu = Relu(%bn3a_branch2a)
  %res3a_branch2b = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%res3a_branch2a_relu, %res3a_branch2b_W, %res3a_branch2b_B)
  %bn3a_branch2b = BatchNormalization[epsilon = 9.99999974737875e-06](%res3a_branch2b, %bn3a_branch2b_scale, %bn3a_branch2b_B, %bn3a_branch2b_mean, %bn3a_branch2b_var)
  %res3a_branch1 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [2, 2]](%res2b_relu, %res3a_branch1_W, %res3a_branch1_B)
  %bn3a_branch1 = BatchNormalization[epsilon = 9.99999974737875e-06](%res3a_branch1, %bn3a_branch1_scale, %bn3a_branch1_B, %bn3a_branch1_mean, %bn3a_branch1_var)
  %res3a = Sum(%bn3a_branch2b, %bn3a_branch1)
  %res3a_relu = Relu(%res3a)
  %res3b_branch2a = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%res3a_relu, %res3b_branch2a_W, %res3b_branch2a_B)
  %bn3b_branch2a = BatchNormalization[epsilon = 9.99999974737875e-06](%res3b_branch2a, %bn3b_branch2a_scale, %bn3b_branch2a_B, %bn3b_branch2a_mean, %bn3b_branch2a_var)
  %res3b_branch2a_relu = Relu(%bn3b_branch2a)
  %res3b_branch2b = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%res3b_branch2a_relu, %res3b_branch2b_W, %res3b_branch2b_B)
  %bn3b_branch2b = BatchNormalization[epsilon = 9.99999974737875e-06](%res3b_branch2b, %bn3b_branch2b_scale, %bn3b_branch2b_B, %bn3b_branch2b_mean, %bn3b_branch2b_var)
  %res3b = Sum(%bn3b_branch2b, %res3a_relu)
  %res3b_relu = Relu(%res3b)
  %res4a_branch2a = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [0, 0, 1, 1], strides = [2, 2]](%res3b_relu, %res4a_branch2a_W, %res4a_branch2a_B)
  %bn4a_branch2a = BatchNormalization[epsilon = 9.99999974737875e-06](%res4a_branch2a, %bn4a_branch2a_scale, %bn4a_branch2a_B, %bn4a_branch2a_mean, %bn4a_branch2a_var)
  %res4a_branch2a_relu = Relu(%bn4a_branch2a)
  %res4a_branch2b = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%res4a_branch2a_relu, %res4a_branch2b_W, %res4a_branch2b_B)
  %bn4a_branch2b = BatchNormalization[epsilon = 9.99999974737875e-06](%res4a_branch2b, %bn4a_branch2b_scale, %bn4a_branch2b_B, %bn4a_branch2b_mean, %bn4a_branch2b_var)
  %res4a_branch1 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [2, 2]](%res3b_relu, %res4a_branch1_W, %res4a_branch1_B)
  %bn4a_branch1 = BatchNormalization[epsilon = 9.99999974737875e-06](%res4a_branch1, %bn4a_branch1_scale, %bn4a_branch1_B, %bn4a_branch1_mean, %bn4a_branch1_var)
  %res4a = Sum(%bn4a_branch2b, %bn4a_branch1)
  %res4a_relu = Relu(%res4a)
  %res4b_branch2a = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%res4a_relu, %res4b_branch2a_W, %res4b_branch2a_B)
  %bn4b_branch2a = BatchNormalization[epsilon = 9.99999974737875e-06](%res4b_branch2a, %bn4b_branch2a_scale, %bn4b_branch2a_B, %bn4b_branch2a_mean, %bn4b_branch2a_var)
  %res4b_branch2a_relu = Relu(%bn4b_branch2a)
  %res4b_branch2b = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%res4b_branch2a_relu, %res4b_branch2b_W, %res4b_branch2b_B)
  %bn4b_branch2b = BatchNormalization[epsilon = 9.99999974737875e-06](%res4b_branch2b, %bn4b_branch2b_scale, %bn4b_branch2b_B, %bn4b_branch2b_mean, %bn4b_branch2b_var)
  %res4b = Sum(%bn4b_branch2b, %res4a_relu)
  %res4b_relu = Relu(%res4b)
  %res5a_branch2a = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%res4b_relu, %res5a_branch2a_W, %res5a_branch2a_B)
  %bn5a_branch2a = BatchNormalization[epsilon = 9.99999974737875e-06](%res5a_branch2a, %bn5a_branch2a_scale, %bn5a_branch2a_B, %bn5a_branch2a_mean, %bn5a_branch2a_var)
  %res5a_branch2a_relu = Relu(%bn5a_branch2a)
  %res5a_branch2b = Conv[dilations = [2, 2], group = 1, kernel_shape = [3, 3], pads = [2, 2, 2, 2], strides = [1, 1]](%res5a_branch2a_relu, %res5a_branch2b_W, %res5a_branch2b_B)
  %bn5a_branch2b = BatchNormalization[epsilon = 9.99999974737875e-06](%res5a_branch2b, %bn5a_branch2b_scale, %bn5a_branch2b_B, %bn5a_branch2b_mean, %bn5a_branch2b_var)
  %res5a_branch1 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%res4b_relu, %res5a_branch1_W, %res5a_branch1_B)
  %bn5a_branch1 = BatchNormalization[epsilon = 9.99999974737875e-06](%res5a_branch1, %bn5a_branch1_scale, %bn5a_branch1_B, %bn5a_branch1_mean, %bn5a_branch1_var)
  %res5a = Sum(%bn5a_branch2b, %bn5a_branch1)
  %res5a_relu = Relu(%res5a)
  %res5b_branch2a = Conv[dilations = [2, 2], group = 1, kernel_shape = [3, 3], pads = [2, 2, 2, 2], strides = [1, 1]](%res5a_relu, %res5b_branch2a_W, %res5b_branch2a_B)
  %bn5b_branch2a = BatchNormalization[epsilon = 9.99999974737875e-06](%res5b_branch2a, %bn5b_branch2a_scale, %bn5b_branch2a_B, %bn5b_branch2a_mean, %bn5b_branch2a_var)
  %res5b_branch2a_relu = Relu(%bn5b_branch2a)
  %res5b_branch2b = Conv[dilations = [2, 2], group = 1, kernel_shape = [3, 3], pads = [2, 2, 2, 2], strides = [1, 1]](%res5b_branch2a_relu, %res5b_branch2b_W, %res5b_branch2b_B)
  %bn5b_branch2b = BatchNormalization[epsilon = 9.99999974737875e-06](%res5b_branch2b, %bn5b_branch2b_scale, %bn5b_branch2b_B, %bn5b_branch2b_mean, %bn5b_branch2b_var)
  %res5b = Sum(%bn5b_branch2b, %res5a_relu)
  %res5b_relu = Relu(%res5b)
  %aspp_Conv_1 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%res5b_relu, %aspp_Conv_1_W, %aspp_Conv_1_B)
  %aspp_BatchNorm_1 = BatchNormalization[epsilon = 9.99999974737875e-06](%aspp_Conv_1, %aspp_BatchNorm_1_scale, %aspp_BatchNorm_1_B, %aspp_BatchNorm_1_mean, %aspp_BatchNorm_1_var)
  %aspp_Relu_1 = Relu(%aspp_BatchNorm_1)
  %aspp_Conv_2 = Conv[dilations = [6, 6], group = 1, kernel_shape = [3, 3], pads = [6, 6, 6, 6], strides = [1, 1]](%res5b_relu, %aspp_Conv_2_W, %aspp_Conv_2_B)
  %aspp_BatchNorm_2 = BatchNormalization[epsilon = 9.99999974737875e-06](%aspp_Conv_2, %aspp_BatchNorm_2_scale, %aspp_BatchNorm_2_B, %aspp_BatchNorm_2_mean, %aspp_BatchNorm_2_var)
  %aspp_Relu_2 = Relu(%aspp_BatchNorm_2)
  %aspp_Conv_3 = Conv[dilations = [12, 12], group = 1, kernel_shape = [3, 3], pads = [12, 12, 12, 12], strides = [1, 1]](%res5b_relu, %aspp_Conv_3_W, %aspp_Conv_3_B)
  %aspp_BatchNorm_3 = BatchNormalization[epsilon = 9.99999974737875e-06](%aspp_Conv_3, %aspp_BatchNorm_3_scale, %aspp_BatchNorm_3_B, %aspp_BatchNorm_3_mean, %aspp_BatchNorm_3_var)
  %aspp_Relu_3 = Relu(%aspp_BatchNorm_3)
  %aspp_Conv_4 = Conv[dilations = [18, 18], group = 1, kernel_shape = [3, 3], pads = [18, 18, 18, 18], strides = [1, 1]](%res5b_relu, %aspp_Conv_4_W, %aspp_Conv_4_B)
  %aspp_BatchNorm_4 = BatchNormalization[epsilon = 9.99999974737875e-06](%aspp_Conv_4, %aspp_BatchNorm_4_scale, %aspp_BatchNorm_4_B, %aspp_BatchNorm_4_mean, %aspp_BatchNorm_4_var)
  %aspp_Relu_4 = Relu(%aspp_BatchNorm_4)
  %catAspp = Concat[axis = 1](%aspp_Relu_1, %aspp_Relu_2, %aspp_Relu_3, %aspp_Relu_4)
  %dec_c1 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%catAspp, %dec_c1_W, %dec_c1_B)
  %dec_bn1 = BatchNormalization[epsilon = 9.99999974737875e-06](%dec_c1, %dec_bn1_scale, %dec_bn1_B, %dec_bn1_mean, %dec_bn1_var)
  %dec_relu1 = Relu(%dec_bn1)
  %dec_upsample1 = ConvTranspose[dilations = [1, 1], group = 1, kernel_shape = [8, 8], pads = [2, 2, 2, 2], strides = [4, 4]](%dec_relu1, %dec_upsample1_W, %dec_upsample1_B)
  %dec_c2 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%res2b_relu, %dec_c2_W, %dec_c2_B)
  %dec_bn2 = BatchNormalization[epsilon = 9.99999974737875e-06](%dec_c2, %dec_bn2_scale, %dec_bn2_B, %dec_bn2_mean, %dec_bn2_var)
  %dec_relu2 = Relu(%dec_bn2)
  %dec_crop1 = Slice[ends = [2147483647, 2147483647, 2147483647, 2147483647], starts = [0, 0, 0, 0]](%dec_upsample1)
  %dec_cat1 = Concat[axis = 1](%dec_relu2, %dec_crop1)
  %dec_c3 = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%dec_cat1, %dec_c3_W, %dec_c3_B)
  %dec_bn3 = BatchNormalization[epsilon = 9.99999974737875e-06](%dec_c3, %dec_bn3_scale, %dec_bn3_B, %dec_bn3_mean, %dec_bn3_var)
  %dec_relu3 = Relu(%dec_bn3)
  %dec_c4 = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%dec_relu3, %dec_c4_W, %dec_c4_B)
  %dec_bn4 = BatchNormalization[epsilon = 9.99999974737875e-06](%dec_c4, %dec_bn4_scale, %dec_bn4_B, %dec_bn4_mean, %dec_bn4_var)
  %dec_relu4 = Relu(%dec_bn4)
  %scorer = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%dec_relu4, %scorer_W, %scorer_B)
  %dec_upsample2 = ConvTranspose[dilations = [1, 1], group = 1, kernel_shape = [8, 8], pads = [2, 2, 2, 2], strides = [4, 4]](%scorer, %dec_upsample2_W, %dec_upsample2_B)
  %dec_crop2 = Slice[ends = [2147483647, 2147483647, 2147483647, 2147483647], starts = [0, 0, 0, 0]](%dec_upsample2)
  %softmax_out_Transpose1 = Transpose[perm = [0, 2, 3, 1]](%dec_crop2)
  %softmax_out = Softmax[axis = 3](%softmax_out_Transpose1)
  %softmax_out_Transpose2 = Transpose[perm = [0, 3, 1, 2]](%softmax_out)
  return %softmax_out_Transpose2
}

[dusty-nv]

Hi @EulersBoiler, I'm not sure about your model, but the way that I export these to ONNX from PyTorch, it is with a fixed input size: https://github.com/dusty-nv/pytorch-segmentation/blob/6ae33ba3426665b8bde8923c0434d207ea4331e0/onnx_export.py#L54

[EulersBoiler]

Alright so i found a solution. I export my models from Matlab which makes two differences to your models in the first layer that one can corrected with the onnx module for python.

[EulersBoiler]

For completeness sake here the solution. The problem is that the onnx is exported with a BATHSIZE parameter in the first layer from which all consecutive layers inherit, therby being dynamic. The first layer has 4 dimensions, i.e. BATCHSIZEx3x224x224 for RGB segmentation. Changing the first dimension to 1 however does not solve the problem, as the dimension is a dim_param and not a dim_value as the other three dimensions. The solution is therefore the one below.

import onnx

model = onnx.load( onnxFilePath )
model.graph.input[0].type.tensor_type.shape.dim[0].dim_value = 1
onnx.save(model, targetFilePath)