search

dusty-nv / jetson-inference

Hello AI World guide to deploying deep-learning inference networks and deep vision primitives with TensorRT and NVIDIA Jetson.
https://developer.nvidia.com/embedded/twodaystoademo
MIT License
7.78k stars 2.98k forks source link

Segnet failed to load network after re-train the segmentation networks #737

Closed Jingchensun closed 1 year ago

Jingchensun commented 4 years ago

I re-trained the segmentation network as the structions in #724 , however there is only four choice for the backbone ( 'deeplabv3_resnet101', 'deeplabv3_resnet50', 'fcn_resnet101', 'fcn_resnet50'), I then choosed fcn_resnet50 to train my dataset.
After I converted the best-model trained to the ONNX model, however, my Jetson Nano seems to could not load the onnx model. Here is the compile:

./segnet.py --camera=/dev/video0 --model=/home/jingchen/jetson-inference/examples/segnet/fcn_resnet50.onnx --width=640 --height=480 --labels=/home/jingchen/jetson-inference/examples/segnet/classes.txt --colors=/home/jingchen/jetson-inference/examples/segnet/colors.txt --input_blob="input_0" --output_blob="output_0"

And here is the bug: `segNet -- loading segmentation network model from: -- prototxt: (null) -- model: /home/jingchen/jetson-inference/examples/segnet/fcn_resnet50.onnx -- labels: /home/jingchen/jetson-inference/examples/segnet/classes.txt -- colors: /home/jingchen/jetson-inference/examples/segnet/colors.txt -- input_blob 'input_0' -- output_blob 'output_0' -- batch_size 1

[TRT] TensorRT version 7.1.3 [TRT] loading NVIDIA plugins... [TRT] Registered plugin creator - ::GridAnchor_TRT version 1 [TRT] Registered plugin creator - ::NMS_TRT version 1 [TRT] Registered plugin creator - ::Reorg_TRT version 1 [TRT] Registered plugin creator - ::Region_TRT version 1 [TRT] Registered plugin creator - ::Clip_TRT version 1 [TRT] Registered plugin creator - ::LReLU_TRT version 1 [TRT] Registered plugin creator - ::PriorBox_TRT version 1 [TRT] Registered plugin creator - ::Normalize_TRT version 1 [TRT] Registered plugin creator - ::RPROI_TRT version 1 [TRT] Registered plugin creator - ::BatchedNMS_TRT version 1 [TRT] Could not register plugin creator - ::FlattenConcat_TRT version 1 [TRT] Registered plugin creator - ::CropAndResize version 1 [TRT] Registered plugin creator - ::DetectionLayer_TRT version 1 [TRT] Registered plugin creator - ::Proposal version 1 [TRT] Registered plugin creator - ::ProposalLayer_TRT version 1 [TRT] Registered plugin creator - ::PyramidROIAlign_TRT version 1 [TRT] Registered plugin creator - ::ResizeNearest_TRT version 1 [TRT] Registered plugin creator - ::Split version 1 [TRT] Registered plugin creator - ::SpecialSlice_TRT version 1 [TRT] Registered plugin creator - ::InstanceNormalization_TRT version 1 [TRT] detected model format - ONNX (extension '.onnx') [TRT] desired precision specified for GPU: FASTEST [TRT] requested fasted precision for device GPU without providing valid calibrator, disabling INT8 [TRT] native precisions detected for GPU: FP32, FP16 [TRT] selecting fastest native precision for GPU: FP16 [TRT] attempting to open engine cache file /home/jingchen/jetson-inference/examples/segnet/fcn_resnet50.onnx.1.1.7103.GPU.FP16.engine [TRT] cache file not found, profiling network model on device GPU [TRT] device GPU, loading /usr/bin/ /home/jingchen/jetson-inference/examples/segnet/fcn_resnet50.onnx

Input filename: /home/jingchen/jetson-inference/examples/segnet/fcn_resnet50.onnx ONNX IR version: 0.0.6 Opset version: 9 Producer name: pytorch Producer version: 1.6 Domain:
Model version: 0 Doc string:

[TRT] Plugin creator already registered - ::GridAnchor_TRT version 1 [TRT] Plugin creator already registered - ::NMS_TRT version 1 [TRT] Plugin creator already registered - ::Reorg_TRT version 1 [TRT] Plugin creator already registered - ::Region_TRT version 1 [TRT] Plugin creator already registered - ::Clip_TRT version 1 [TRT] Plugin creator already registered - ::LReLU_TRT version 1 [TRT] Plugin creator already registered - ::PriorBox_TRT version 1 [TRT] Plugin creator already registered - ::Normalize_TRT version 1 [TRT] Plugin creator already registered - ::RPROI_TRT version 1 [TRT] Plugin creator already registered - ::BatchedNMS_TRT version 1 [TRT] Could not register plugin creator - ::FlattenConcat_TRT version 1 [TRT] Plugin creator already registered - ::CropAndResize version 1 [TRT] Plugin creator already registered - ::DetectionLayer_TRT version 1 [TRT] Plugin creator already registered - ::Proposal version 1 [TRT] Plugin creator already registered - ::ProposalLayer_TRT version 1 [TRT] Plugin creator already registered - ::PyramidROIAlign_TRT version 1 [TRT] Plugin creator already registered - ::ResizeNearest_TRT version 1 [TRT] Plugin creator already registered - ::Split version 1 [TRT] Plugin creator already registered - ::SpecialSlice_TRT version 1 [TRT] Plugin creator already registered - ::InstanceNormalization_TRT version 1 [TRT] ModelImporter.cpp:202: Adding network input: input_0 with dtype: float32, dimensions: (1, 3, 320, 320) [TRT] ImporterContext.hpp:116: Registering tensor: input_0 for ONNX tensor: input_0 [TRT] ModelImporter.cpp:90: Importing initializer: backbone.bn1.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.bn1.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.bn1.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.bn1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.conv1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.0.bn1.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.0.bn1.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.0.bn1.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.0.bn1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.0.bn2.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.0.bn2.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.0.bn2.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.0.bn2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.0.bn3.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.0.bn3.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.0.bn3.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.0.bn3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.0.conv1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.0.conv2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.0.conv3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.0.downsample.0.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.0.downsample.1.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.0.downsample.1.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.0.downsample.1.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.0.downsample.1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.1.bn1.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.1.bn1.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.1.bn1.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.1.bn1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.1.bn2.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.1.bn2.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.1.bn2.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.1.bn2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.1.bn3.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.1.bn3.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.1.bn3.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.1.bn3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.1.conv1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.1.conv2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.1.conv3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.2.bn1.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.2.bn1.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.2.bn1.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.2.bn1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.2.bn2.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.2.bn2.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.2.bn2.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.2.bn2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.2.bn3.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.2.bn3.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.2.bn3.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.2.bn3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.2.conv1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.2.conv2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer1.2.conv3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.0.bn1.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.0.bn1.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.0.bn1.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.0.bn1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.0.bn2.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.0.bn2.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.0.bn2.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.0.bn2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.0.bn3.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.0.bn3.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.0.bn3.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.0.bn3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.0.conv1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.0.conv2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.0.conv3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.0.downsample.0.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.0.downsample.1.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.0.downsample.1.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.0.downsample.1.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.0.downsample.1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.1.bn1.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.1.bn1.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.1.bn1.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.1.bn1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.1.bn2.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.1.bn2.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.1.bn2.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.1.bn2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.1.bn3.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.1.bn3.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.1.bn3.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.1.bn3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.1.conv1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.1.conv2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.1.conv3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.2.bn1.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.2.bn1.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.2.bn1.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.2.bn1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.2.bn2.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.2.bn2.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.2.bn2.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.2.bn2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.2.bn3.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.2.bn3.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.2.bn3.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.2.bn3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.2.conv1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.2.conv2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.2.conv3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.3.bn1.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.3.bn1.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.3.bn1.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.3.bn1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.3.bn2.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.3.bn2.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.3.bn2.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.3.bn2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.3.bn3.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.3.bn3.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.3.bn3.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.3.bn3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.3.conv1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.3.conv2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer2.3.conv3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.0.bn1.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.0.bn1.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.0.bn1.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.0.bn1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.0.bn2.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.0.bn2.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.0.bn2.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.0.bn2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.0.bn3.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.0.bn3.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.0.bn3.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.0.bn3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.0.conv1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.0.conv2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.0.conv3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.0.downsample.0.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.0.downsample.1.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.0.downsample.1.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.0.downsample.1.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.0.downsample.1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.1.bn1.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.1.bn1.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.1.bn1.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.1.bn1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.1.bn2.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.1.bn2.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.1.bn2.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.1.bn2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.1.bn3.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.1.bn3.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.1.bn3.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.1.bn3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.1.conv1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.1.conv2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.1.conv3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.2.bn1.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.2.bn1.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.2.bn1.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.2.bn1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.2.bn2.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.2.bn2.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.2.bn2.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.2.bn2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.2.bn3.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.2.bn3.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.2.bn3.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.2.bn3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.2.conv1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.2.conv2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.2.conv3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.3.bn1.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.3.bn1.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.3.bn1.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.3.bn1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.3.bn2.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.3.bn2.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.3.bn2.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.3.bn2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.3.bn3.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.3.bn3.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.3.bn3.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.3.bn3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.3.conv1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.3.conv2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.3.conv3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.4.bn1.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.4.bn1.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.4.bn1.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.4.bn1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.4.bn2.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.4.bn2.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.4.bn2.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.4.bn2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.4.bn3.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.4.bn3.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.4.bn3.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.4.bn3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.4.conv1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.4.conv2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.4.conv3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.5.bn1.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.5.bn1.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.5.bn1.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.5.bn1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.5.bn2.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.5.bn2.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.5.bn2.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.5.bn2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.5.bn3.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.5.bn3.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.5.bn3.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.5.bn3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.5.conv1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.5.conv2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer3.5.conv3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.0.bn1.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.0.bn1.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.0.bn1.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.0.bn1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.0.bn2.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.0.bn2.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.0.bn2.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.0.bn2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.0.bn3.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.0.bn3.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.0.bn3.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.0.bn3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.0.conv1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.0.conv2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.0.conv3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.0.downsample.0.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.0.downsample.1.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.0.downsample.1.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.0.downsample.1.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.0.downsample.1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.1.bn1.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.1.bn1.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.1.bn1.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.1.bn1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.1.bn2.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.1.bn2.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.1.bn2.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.1.bn2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.1.bn3.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.1.bn3.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.1.bn3.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.1.bn3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.1.conv1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.1.conv2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.1.conv3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.2.bn1.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.2.bn1.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.2.bn1.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.2.bn1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.2.bn2.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.2.bn2.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.2.bn2.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.2.bn2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.2.bn3.bias [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.2.bn3.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.2.bn3.running_var [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.2.bn3.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.2.conv1.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.2.conv2.weight [TRT] ModelImporter.cpp:90: Importing initializer: backbone.layer4.2.conv3.weight [TRT] ModelImporter.cpp:90: Importing initializer: classifier.0.weight [TRT] ModelImporter.cpp:90: Importing initializer: classifier.1.bias [TRT] ModelImporter.cpp:90: Importing initializer: classifier.1.running_mean [TRT] ModelImporter.cpp:90: Importing initializer: classifier.1.running_var [TRT] ModelImporter.cpp:90: Importing initializer: classifier.1.weight [TRT] ModelImporter.cpp:90: Importing initializer: classifier.4.bias [TRT] ModelImporter.cpp:90: Importing initializer: classifier.4.weight [TRT] ModelImporter.cpp:103: Parsing node: Shape_0 [Shape] [TRT] ModelImporter.cpp:119: Searching for input: input_0 [TRT] ModelImporter.cpp:125: Shape_0 [Shape] inputs: [input_0 -> (1, 3, 320, 320)], [TRT] ImporterContext.hpp:141: Registering layer: Shape_0 for ONNX node: Shape_0 [TRT] ImporterContext.hpp:116: Registering tensor: 327 for ONNX tensor: 327 [TRT] ModelImporter.cpp:179: Shape_0 [Shape] outputs: [327 -> (4)], [TRT] ModelImporter.cpp:103: Parsing node: Constant_1 [Constant] [TRT] ModelImporter.cpp:125: Constant_1 [Constant] inputs: [TRT] onnx2trt_utils.cpp:220: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TRT] ModelImporter.cpp:179: Constant_1 [Constant] outputs: [328 -> ()], [TRT] ModelImporter.cpp:103: Parsing node: Gather_2 [Gather] [TRT] ModelImporter.cpp:119: Searching for input: 327 [TRT] ModelImporter.cpp:119: Searching for input: 328 [TRT] ModelImporter.cpp:125: Gather_2 [Gather] inputs: [327 -> (4)], [328 -> ()], [TRT] builtin_op_importers.cpp:986: Using Gather axis: 0 [TRT] ImporterContext.hpp:141: Registering layer: Gather_2 for ONNX node: Gather_2 [TRT] ImporterContext.hpp:116: Registering tensor: 329 for ONNX tensor: 329 [TRT] ModelImporter.cpp:179: Gather_2 [Gather] outputs: [329 -> ()], [TRT] ModelImporter.cpp:103: Parsing node: Shape_3 [Shape] [TRT] ModelImporter.cpp:119: Searching for input: input_0 [TRT] ModelImporter.cpp:125: Shape_3 [Shape] inputs: [input_0 -> (1, 3, 320, 320)], [TRT] ImporterContext.hpp:141: Registering layer: Shape_3 for ONNX node: Shape_3 [TRT] ImporterContext.hpp:116: Registering tensor: 330 for ONNX tensor: 330 [TRT] ModelImporter.cpp:179: Shape_3 [Shape] outputs: [330 -> (4)], [TRT] ModelImporter.cpp:103: Parsing node: Constant_4 [Constant] [TRT] ModelImporter.cpp:125: Constant_4 [Constant] inputs: [TRT] ModelImporter.cpp:179: Constant_4 [Constant] outputs: [331 -> ()], [TRT] ModelImporter.cpp:103: Parsing node: Gather_5 [Gather] [TRT] ModelImporter.cpp:119: Searching for input: 330 [TRT] ModelImporter.cpp:119: Searching for input: 331 [TRT] ModelImporter.cpp:125: Gather_5 [Gather] inputs: [330 -> (4)], [331 -> ()], [TRT] builtin_op_importers.cpp:986: Using Gather axis: 0 [TRT] ImporterContext.hpp:141: Registering layer: Gather_5 for ONNX node: Gather_5 [TRT] ImporterContext.hpp:116: Registering tensor: 332 for ONNX tensor: 332 [TRT] ModelImporter.cpp:179: Gather_5 [Gather] outputs: [332 -> ()], [TRT] ModelImporter.cpp:103: Parsing node: Conv_6 [Conv] [TRT] ModelImporter.cpp:119: Searching for input: input_0 [TRT] ModelImporter.cpp:119: Searching for input: backbone.conv1.weight [TRT] ModelImporter.cpp:125: Conv_6 [Conv] inputs: [input_0 -> (1, 3, 320, 320)], [backbone.conv1.weight -> (64, 3, 7, 7)], [TRT] builtin_op_importers.cpp:450: Convolution input dimensions: (1, 3, 320, 320) [TRT] ImporterContext.hpp:141: Registering layer: Conv_6 for ONNX node: Conv_6 [TRT] builtin_op_importers.cpp:533: Using kernel: (7, 7), strides: (2, 2), prepadding: (3, 3), postpadding: (3, 3), dilations: (1, 1), numOutputs: 64 [TRT] builtin_op_importers.cpp:534: Convolution output dimensions: (1, 64, 160, 160) [TRT] ImporterContext.hpp:116: Registering tensor: 333 for ONNX tensor: 333 [TRT] ModelImporter.cpp:179: Conv_6 [Conv] outputs: [333 -> (1, 64, 160, 160)], [TRT] ModelImporter.cpp:103: Parsing node: BatchNormalization_7 [BatchNormalization] [TRT] ModelImporter.cpp:119: Searching for input: 333 [TRT] ModelImporter.cpp:119: Searching for input: backbone.bn1.weight [TRT] ModelImporter.cpp:119: Searching for input: backbone.bn1.bias [TRT] ModelImporter.cpp:119: Searching for input: backbone.bn1.running_mean [TRT] ModelImporter.cpp:119: Searching for input: backbone.bn1.running_var [TRT] ModelImporter.cpp:125: BatchNormalization_7 [BatchNormalization] inputs: [333 -> (1, 64, 160, 160)], [backbone.bn1.weight -> (64)], [backbone.bn1.bias -> (64)], [backbone.bn1.running_mean -> (64)], [backbone.bn1.running_var -> (64)],

......

[TRT] builtin_op_importers.cpp:534: Convolution output dimensions: (1, 2, 40, 40) [TRT] ImporterContext.hpp:116: Registering tensor: 508 for ONNX tensor: 508 [TRT] ModelImporter.cpp:179: Conv_181 [Conv] outputs: [508 -> (1, 2, 40, 40)], [TRT] ModelImporter.cpp:103: Parsing node: Unsqueeze_182 [Unsqueeze] [TRT] ModelImporter.cpp:119: Searching for input: 329 [TRT] ModelImporter.cpp:125: Unsqueeze_182 [Unsqueeze] inputs: [329 -> ()], [TRT] onnx2trt_utils.cpp:1793: Original shape: (), unsqueezing to: (1,) [TRT] ImporterContext.hpp:141: Registering layer: Unsqueeze_182 for ONNX node: Unsqueeze_182 [TRT] ImporterContext.hpp:116: Registering tensor: 509 for ONNX tensor: 509 [TRT] ModelImporter.cpp:179: Unsqueeze_182 [Unsqueeze] outputs: [509 -> (1)], [TRT] ModelImporter.cpp:103: Parsing node: Unsqueeze_183 [Unsqueeze] [TRT] ModelImporter.cpp:119: Searching for input: 332 [TRT] ModelImporter.cpp:125: Unsqueeze_183 [Unsqueeze] inputs: [332 -> ()], [TRT] onnx2trt_utils.cpp:1793: Original shape: (), unsqueezing to: (1,) [TRT] ImporterContext.hpp:141: Registering layer: Unsqueeze_183 for ONNX node: Unsqueeze_183 [TRT] ImporterContext.hpp:116: Registering tensor: 510 for ONNX tensor: 510 [TRT] ModelImporter.cpp:179: Unsqueeze_183 [Unsqueeze] outputs: [510 -> (1)], [TRT] ModelImporter.cpp:103: Parsing node: Concat_184 [Concat] [TRT] ModelImporter.cpp:119: Searching for input: 509 [TRT] ModelImporter.cpp:119: Searching for input: 510 [TRT] ModelImporter.cpp:125: Concat_184 [Concat] inputs: [509 -> (1)], [510 -> (1)], [TRT] ImporterContext.hpp:141: Registering layer: Concat_184 for ONNX node: Concat_184 [TRT] ImporterContext.hpp:116: Registering tensor: 511 for ONNX tensor: 511 [TRT] ModelImporter.cpp:179: Concat_184 [Concat] outputs: [511 -> (2)], [TRT] ModelImporter.cpp:103: Parsing node: Constant_185 [Constant] [TRT] ModelImporter.cpp:125: Constant_185 [Constant] inputs: [TRT] ModelImporter.cpp:179: Constant_185 [Constant] outputs: [512 -> (2)], [TRT] ModelImporter.cpp:103: Parsing node: Cast_186 [Cast] [TRT] ModelImporter.cpp:119: Searching for input: 511 [TRT] ModelImporter.cpp:125: Cast_186 [Cast] inputs: [511 -> (2)], [TRT] builtin_op_importers.cpp:320: Casting to type: float32 [TRT] ImporterContext.hpp:141: Registering layer: Cast_186 for ONNX node: Cast_186 [TRT] ImporterContext.hpp:116: Registering tensor: 513 for ONNX tensor: 513 [TRT] ModelImporter.cpp:179: Cast_186 [Cast] outputs: [513 -> (2)], [TRT] ModelImporter.cpp:103: Parsing node: Shape_187 [Shape] [TRT] ModelImporter.cpp:119: Searching for input: 508 [TRT] ModelImporter.cpp:125: Shape_187 [Shape] inputs: [508 -> (1, 2, 40, 40)], [TRT] ImporterContext.hpp:141: Registering layer: Shape_187 for ONNX node: Shape_187 [TRT] ImporterContext.hpp:116: Registering tensor: 514 for ONNX tensor: 514 [TRT] ModelImporter.cpp:179: Shape_187 [Shape] outputs: [514 -> (4)], [TRT] ModelImporter.cpp:103: Parsing node: Slice_188 [Slice] [TRT] ModelImporter.cpp:119: Searching for input: 514 [TRT] ModelImporter.cpp:125: Slice_188 [Slice] inputs: [514 -> (4)], [TRT] ImporterContext.hpp:141: Registering layer: Slice_188 for ONNX node: Slice_188 [TRT] ImporterContext.hpp:116: Registering tensor: 515 for ONNX tensor: 515 [TRT] ModelImporter.cpp:179: Slice_188 [Slice] outputs: [515 -> (2)], [TRT] ModelImporter.cpp:103: Parsing node: Cast_189 [Cast] [TRT] ModelImporter.cpp:119: Searching for input: 515 [TRT] ModelImporter.cpp:125: Cast_189 [Cast] inputs: [515 -> (2)], [TRT] builtin_op_importers.cpp:320: Casting to type: float32 [TRT] ImporterContext.hpp:141: Registering layer: Cast_189 for ONNX node: Cast_189 [TRT] ImporterContext.hpp:116: Registering tensor: 516 for ONNX tensor: 516 [TRT] ModelImporter.cpp:179: Cast_189 [Cast] outputs: [516 -> (2)], [TRT] ModelImporter.cpp:103: Parsing node: Div_190 [Div] [TRT] ModelImporter.cpp:119: Searching for input: 513 [TRT] ModelImporter.cpp:119: Searching for input: 516 [TRT] ModelImporter.cpp:125: Div_190 [Div] inputs: [513 -> (2)], [516 -> (2)], [TRT] ImporterContext.hpp:141: Registering layer: Div_190 for ONNX node: Div_190 [TRT] ImporterContext.hpp:116: Registering tensor: 517 for ONNX tensor: 517 [TRT] ModelImporter.cpp:179: Div_190 [Div] outputs: [517 -> (2)], [TRT] ModelImporter.cpp:103: Parsing node: Concat_191 [Concat] [TRT] ModelImporter.cpp:119: Searching for input: 512 [TRT] ModelImporter.cpp:119: Searching for input: 517 [TRT] ModelImporter.cpp:125: Concat_191 [Concat] inputs: [512 -> (2)], [517 -> (2)], [TRT] ImporterContext.hpp:141: Registering layer: Concat_191 for ONNX node: Concat_191 [TRT] ImporterContext.hpp:116: Registering tensor: 518 for ONNX tensor: 518 [TRT] ModelImporter.cpp:179: Concat_191 [Concat] outputs: [518 -> (4)], [TRT] ModelImporter.cpp:103: Parsing node: Upsample_192 [Upsample] [TRT] ModelImporter.cpp:119: Searching for input: 508 [TRT] ModelImporter.cpp:119: Searching for input: 518 [TRT] ModelImporter.cpp:125: Upsample_192 [Upsample] inputs: [508 -> (1, 2, 40, 40)], [518 -> (4)], ERROR: builtin_op_importers.cpp:3460 In function importUpsample: [8] Assertion failed: scales_input.is_weights() [TRT] failed to parse ONNX model '/home/jingchen/jetson-inference/examples/segnet/fcn_resnet50.onnx' [TRT] device GPU, failed to load /home/jingchen/jetson-inference/examples/segnet/fcn_resnet50.onnx [TRT] segNet -- failed to load. jetson.inference -- segNet failed to load network Traceback (most recent call last): File "segnet.py", line 56, in net = jetson.inference.segNet(opt.network, sys.argv) Exception: jetson.inference -- segNet failed to load network`

Is this because the fcn_resnet50 models too heavy that Jetson Nano can't load ?

And By the Way, as mentioned in #437 , could you help me how to download the default resnet-based FCN models since the mirror downloads only provide the alexnet-based FCN models

dusty-nv commented 4 years ago

Hi @Jingchensun , my fork of torchvision (0.3.0) branch contains the support for FCN-ResNet18 and also modifications to make it able to be loaded in TensorRT:

https://github.com/dusty-nv/vision/tree/v0.3.0

There is also a segmentation training tutorial here: https://www.highvoltagecode.com/post/edge-ai-semantic-segmentation-on-nvidia-jetson

dusty-nv commented 4 years ago

I've also uploaded the FCN-ResNet18 models to the mirror page here: https://github.com/dusty-nv/jetson-inference/releases/tag/model-mirror-190618

Onixaz commented 4 years ago

@Jingchensun if you followed my tutorial, did you use "pip install git+https://github.com/Onixaz/vision@v0.3.0" to install torchvision library ?

@dusty-nv I actually made a fix in your branch, mentioned here https://github.com/dusty-nv/pytorch-segmentation/issues/2#issuecomment-576907769

Could make a pull request, but then again I'm not sure what causes this....

dusty-nv commented 4 years ago

What I would like to do in the longer term is to take our customizations from torchvision and just move them into pytorch-segmentation, so that pytorch-segmentation repo does not require the customized fork of torchvision.

Happy to team up on it @Onixaz if you are interested, although I am busy with other tasks the next few weeks. Ideally I would like to include segmentation training tutorial in Hello AI World, but I don't know if it would run on Jetson or not due to memory/compute resources.

Onixaz commented 4 years ago

That would be ideal. Sure @dusty-nv , I'm on a strict schedule myself, but I'll start looking into it.

marvision-ai commented 4 years ago

@Onixaz and @dusty-nv Thank you very much for the segmentation repo and the great tutorial!

I am on the last step of training the network (prepped data and everything) and run into this issue. Can you pinpoint why? My images are of size 1500x624

$python train.py /home/ai/algorithms/jetson_segmentation/pytorch-segmentation/5Y2_final --dataset=custom --classes=1 --width 1500 --height 624

pytorch-segmentation/datasets/__init__.py
Not using distributed mode
Namespace(arch='fcn_resnet18', aux_loss=False, batch_size=4, classes=1, data='/home/ai/algorithms/jetson_segmentation/pytorch-segmentation/5Y2_final', dataset='custom', device='cuda', dist_url='env://', distributed=False, epochs=30, height=624, lr=0.01, model_dir='.', momentum=0.9, pretrained=False, print_freq=10, resolution=320, resume='', test_only=False, weight_decay=0.0001, width=1500, workers=16, world_size=1)
=> training with dataset: 'custom' (train=460, val=116)
=> training with resolution: 1500x624, 1 classes
=> training with model: fcn_resnet18
torchvision.models.segmentation.fcn_resnet18()
torchvision.models.segmentation.FCN() => configuring model for training
Traceback (most recent call last):
  File "train.py", line 334, in <module>
    main(args)
  File "train.py", line 295, in main
    train_one_epoch(model, criterion, optimizer, data_loader, lr_scheduler, device, epoch, args.print_freq)
  File "train.py", line 192, in train_one_epoch
    loss.backward()
  File "/home/.virtualenvs/jetson_seg/lib/python3.6/site-packages/torch/tensor.py", line 107, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/.virtualenvs/jetson_seg/lib/python3.6/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: CUDA error: device-side assert triggered

Let me know! Thank you in advance :smile:

Onixaz commented 4 years ago

Hi @marvision-ai,

Try --classes=2, since background counts as a "class" as well.

marvision-ai commented 4 years ago

Hi @Onixaz , Yup that was it! I feel so silly. Thanks for the help :+1:

khsafkatamin commented 7 months ago

hi @dusty-nv , I am facing the same issue. I trained a custom segnet model using https://www.highvoltagecode.com/post/edge-ai-semantic-segmentation-on-nvidia-jetson and generated an onnx file with which I can successfully run inference on my local machine using docker.

But when I tried to implement it on Jetson AGX Xavier, I face loading the model. I tried both in docker and locally. below is the bug:

``segNet -- loading segmentation network model from: -- prototxt: (null) -- model: fcn_resnet50.onnx -- labels: classes.txt -- colors: colors.txt -- input_blob 'input_0' -- output_blob 'output_0' -- batch_size 1

[TRT] TensorRT version 8.5.2 [TRT] loading NVIDIA plugins... [TRT] Registered plugin creator - ::BatchedNMSDynamic_TRT version 1 [TRT] Registered plugin creator - ::BatchedNMS_TRT version 1 [TRT] Registered plugin creator - ::BatchTilePlugin_TRT version 1 [TRT] Registered plugin creator - ::Clip_TRT version 1 [TRT] Registered plugin creator - ::CoordConvAC version 1 [TRT] Registered plugin creator - ::CropAndResizeDynamic version 1 [TRT] Registered plugin creator - ::CropAndResize version 1 [TRT] Registered plugin creator - ::DecodeBbox3DPlugin version 1 [TRT] Registered plugin creator - ::DetectionLayer_TRT version 1 [TRT] Registered plugin creator - ::EfficientNMS_Explicit_TF_TRT version 1 [TRT] Registered plugin creator - ::EfficientNMS_Implicit_TF_TRT version 1 [TRT] Registered plugin creator - ::EfficientNMS_ONNX_TRT version 1 [TRT] Registered plugin creator - ::EfficientNMS_TRT version 1 [TRT] Could not register plugin creator - ::FlattenConcat_TRT version 1 [TRT] Registered plugin creator - ::GenerateDetection_TRT version 1 [TRT] Registered plugin creator - ::GridAnchor_TRT version 1 [TRT] Registered plugin creator - ::GridAnchorRect_TRT version 1 [TRT] Registered plugin creator - ::GroupNorm version 1 [TRT] Registered plugin creator - ::InstanceNormalization_TRT version 1 [TRT] Registered plugin creator - ::InstanceNormalization_TRT version 2 [TRT] Registered plugin creator - ::LayerNorm version 1 [TRT] Registered plugin creator - ::LReLU_TRT version 1 [TRT] Registered plugin creator - ::MultilevelCropAndResize_TRT version 1 [TRT] Registered plugin creator - ::MultilevelProposeROI_TRT version 1 [TRT] Registered plugin creator - ::MultiscaleDeformableAttnPlugin_TRT version 1 [TRT] Registered plugin creator - ::NMSDynamic_TRT version 1 [TRT] Registered plugin creator - ::NMS_TRT version 1 [TRT] Registered plugin creator - ::Normalize_TRT version 1 [TRT] Registered plugin creator - ::PillarScatterPlugin version 1 [TRT] Registered plugin creator - ::PriorBox_TRT version 1 [TRT] Registered plugin creator - ::ProposalDynamic version 1 [TRT] Registered plugin creator - ::ProposalLayer_TRT version 1 [TRT] Registered plugin creator - ::Proposal version 1 [TRT] Registered plugin creator - ::PyramidROIAlign_TRT version 1 [TRT] Registered plugin creator - ::Region_TRT version 1 [TRT] Registered plugin creator - ::Reorg_TRT version 1 [TRT] Registered plugin creator - ::ResizeNearest_TRT version 1 [TRT] Registered plugin creator - ::ROIAlign_TRT version 1 [TRT] Registered plugin creator - ::RPROI_TRT version 1 [TRT] Registered plugin creator - ::ScatterND version 1 [TRT] Registered plugin creator - ::SeqLen2Spatial version 1 [TRT] Registered plugin creator - ::SpecialSlice_TRT version 1 [TRT] Registered plugin creator - ::SplitGeLU version 1 [TRT] Registered plugin creator - ::Split version 1 [TRT] Registered plugin creator - ::VoxelGeneratorPlugin version 1 [TRT] completed loading NVIDIA plugins. [TRT] detected model format - ONNX (extension '.onnx') [TRT] desired precision specified for GPU: FASTEST [TRT] requested fasted precision for device GPU without providing valid calibrator, disabling INT8 [TRT] [MemUsageChange] Init CUDA: CPU +188, GPU +0, now: CPU 221, GPU 8477 (MiB) [TRT] Trying to load shared library libnvinfer_builder_resource.so.8.5.2 [TRT] Loaded shared library libnvinfer_builder_resource.so.8.5.2 [TRT] [MemUsageChange] Init builder kernel library: CPU +106, GPU +445, now: CPU 350, GPU 9011 (MiB) [TRT] native precisions detected for GPU: FP32, FP16, INT8 [TRT] selecting fastest native precision for GPU: FP16 [TRT] could not find engine cache fcn_resnet50.onnx.1.1.8502.GPU.FP16.engine [TRT] cache file invalid, profiling network model on device GPU [TRT] [MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 246, GPU 9032 (MiB) [TRT] Trying to load shared library libnvinfer_builder_resource.so.8.5.2 [TRT] Loaded shared library libnvinfer_builder_resource.so.8.5.2 [TRT] [MemUsageChange] Init builder kernel library: CPU +104, GPU +177, now: CPU 350, GPU 9363 (MiB) [TRT] The implicit batch dimension mode has been deprecated. Please create the network with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag whenever possible. [TRT] device GPU, loading /usr/local/bin/ fcn_resnet50.onnx [libprotobuf ERROR google/protobuf/text_format.cc:298] Error parsing text-format onnx2trt_onnx.ModelProto: 1:1: Interpreting non ascii codepoint 175. [libprotobuf ERROR google/protobuf/text_format.cc:298] Error parsing text-format onnx2trt_onnx.ModelProto: 1:1: Expected identifier, got: � [TRT] ModelImporter.cpp:688: Failed to parse ONNX model from file: fcn_resnet50.onnx [TRT] failed to parse ONNX model 'fcn_resnet50.onnx' [TRT] device GPU, failed to load fcn_resnet50.onnx [TRT] segNet -- failed to load. segnet: failed to initialize segNet `` I don't understand what's wrong in here. can you help me?

khsafkatamin commented 7 months ago

Sorry, it was a silly issue. The issue popped up as I tried to load the ONNX file generated in my PC(x86) interface into Jetson. It got sorted when I generated the ONNX format directly on Jetson.

Thanks anyways and great work!