detectnet-camera proper arguments for loading custom model?

[rsamvelyan]

Here's a quick question: If I want to use my custom mobilenet-v1 model with Jetson Inference, in the call for detectnet-camera.py do I still need to specify the --network parameter or do I omit it?

In other words which one of these two is correct?

1- python3 detectnet-camera.py --network=ssd-mobilenet-v1 --model=models/mb1-ssd.onnx --class_labels=models/voc-model-labels.txt

OR

2- python3 detectnet-camera.py --model=models/mb1-ssd.onnx --class_labels=models/voc-model-labels.txt

Thanks!!

[dusty-nv]

Hi @rsamvelyan , you would want to omit --network (or use only one, where the path to your model is the argument).

You would also need to specify --input_blob=input_0, --output_cvg=scores, and --output_bbox=boxes. These are the layer names for SSD set by convert_to_onnx.py

[rsamvelyan]

So if I were to load my own ONNX model, could I use --network=path/to/my/model.onnx? Or just skip that and use --model=path/to/my/model.onnx?

Could you please provide me with a line with all the arguments? ex:

python3 detectnet-camera.py 
--model=models/mb1-ssd.onnx 
--class_labels=models/voc-model-labels.txt
--input_blob=input_0
--output_cvg=scores
--output_bbox=boxes

[dusty-nv]

if I were to load my own ONNX model, could I use --network=path/to/my/model.onnx? Or just skip that and use --model=path/to/my/model.onnx?

You could use either of them, but just pick one. If you specify --network=ssd-mobilenet-v1, that would have overridden your custom model and just loaded the pretrained COCO model.

Could you please provide me with a line with all the arguments? ex:

python3 detectnet-camera.py 
--model=models/mb1-ssd.onnx 
--class_labels=models/voc-model-labels.txt
--input_blob=input_0
--output_cvg=scores
--output_bbox=boxes

I believe that should be it - I am getting back up to speed on my ONNX SSD support since the NX release. You probably want to try detectnet-console on a static test image first. And if you have trouble with parsing the arguments with the python version, try the C++ version.

[rsamvelyan]

I ran the conversion to ONNX and with the parameters below I was able to run inference. 35-38 FPS with an amazing accuracy!!

detectnet-camera --model=mb1-ssd.onnx --class_labels=voc-model-labels.txt --input_blob=input_0 --output_cvg=scores --output_bbox=boxes --camera=/dev/video0 --width=640  --height=480

Thanks a lot Dusty!!!

Now next question, can we do learning transfer on a custom dataset on mobile-net-v1 (this very same model) with Pytorch?

[dusty-nv]

@rsamvelyan I'm glad to hear that you got it working well, I believe you are the first person to test this other than myself, so thanks for bearing with me.

This part of the SSD repo shows how to re-train on Google Open Images dataset: https://github.com/dusty-nv/pytorch-ssd#retrain-on-open-images-dataset

There are many, many classes in Open Images dataset, so you can pick which ones you want to use and substitute those into the commands shown in the repo. You can browse the classes here: https://storage.googleapis.com/openimages/web/visualizer/index.html?set=train&type=detection&c=%2Fm%2F01g317

Although check the open_images_downloader.py script for extra arguments I added to limit the number of images that it can download, because sometimes Open Images will have hundreds of thousands of images in some popular classes (and that would take forever to train on Jetson). So you probably want to limit it to 10-25K or so.

Also, I am currently working on the data capture tool in jetson-inference to add support for object detection. It will save the user's custom dataset in VOC PASCAL format so it can be used with the VOC dataloader already present in pytorch-ssd repo.

[rsamvelyan]

Ok great, I will go through this link you sent me and try out the training.

I already prepared a python script which which goes into the folder containing training or validation images (in JPG format) and annotations (Pascal VOC .xml), validates every single annotation file and puts everything into an array, ready to be fed to Pytorch trainer (dataloader or something?) If it helps, I'll post it.

[tmralmeida]

I ran the conversion to ONNX and with the parameters below I was able to run inference. 35-38 FPS with an amazing accuracy!!

detectnet-camera --model=mb1-ssd.onnx --class_labels=voc-model-labels.txt --input_blob=input_0 --output_cvg=scores --output_bbox=boxes --camera=/dev/video0 --width=640  --height=480

Thanks a lot Dusty!!!

Now next question, can we do learning transfer on a custom dataset on mobile-net-v1 (this very same model) with Pytorch?

Hi!

I did exact the same thing but my custom model is a ssd with moblenetv2 and by running the command:

./detectnet-camera --model=networks/my-ssd-model.onnx --class_labels=networks/bdd-model-labels.txt --input_blob=input_0 --output_cvg=scores  --output_bbox=boxes --camera=/dev/video0

I obtain the error: --- End node --- ERROR: ModelImporter.cpp:288 In function importModel: [5] Assertion failed: tensors.count(input_name) [TRT] failed to parse ONNX model 'networks/my-ssd-model.onnx' [TRT] device GPU, failed to load networks/my-ssd-model.onnx detectNet -- failed to initialize. detectnet-camera: failed to load detectNet model

Any idea? Thanks!!

[tmralmeida]

---

[UPDATE] Hi again @dusty-nv,

I solved the initial issue, however,it seems that there is an incompability between onnx and tensorrt. I used your pytorch-detection repo to train and convert the model to ONNX. I obtained this error:

My tensorrt version is 5.1.6.

Could you help me on that? Thanks!

---

[NEW UPDATE]

After applying onnx-simplifier(https://github.com/daquexian/onnx-simplifier), I've made a little progress and then I got this error (this time related to Slice operation):

Thanks!

---

[NEW UPDATE2]

Ok, I found the problem! It is related to the conversion of regressional location results into boxes in the form of (center_x, center_y, h, w). Concretely, in:

if self.is_test:
            confidences = F.softmax(confidences, dim=2)
            boxes = convert_locations_to_boxes(
                locations, self.priors, self.config.center_variance, self.config.size_variance
            )
            boxes = center_form_to_corner_form(boxes)
            return confidences, boxes

Because, inside of the respective function, there is a:

torch.cat([
        locations[..., :2] * center_variance * priors[..., 2:] + priors[..., :2],
        torch.exp(locations[..., 2:] * size_variance) * priors[..., 2:]
    ], dim=locations.dim() - 1)

If you convert the trained model into ONNX in training mode (i.e. "is_test=False"), there is no error when running the detectnet-camera application. Thus, I'll try to fix this!

[rsamvelyan]

Hi Tmralmedia, I looked at your issue but this is too advanced for me. Do you know by the way the meaning of these parameters during the inference:

--input_blob=input_0 --output_cvg=scores --output_bbox=boxes

[tmralmeida]

Hi @rsamvelyan, those "parameters" are just the names that you gave for the input and output layers when you transformed your model into ONNX. So, the "input_blob" argument is the name of the input layer, the "output_cvg" is the name of the coverage output layer (each confidence in our case) and the "ouput_bbox" is the name of the layer related to the coordinates of the bounding boxes.

[rsamvelyan]

Thanks a lot! Learned something new today :D

Did your ONNX Inference work?

[tmralmeida]

No, it did not work yet just because I've TensorRT 5.1. So, if I flash my Xavier AGX (only way of obtaining newer versions :-1: ) with an upgraded version of tensorRT I think that it will work!

[AiueoABC]

Hi, @tmralmeida I'm trying to use ssd with moblenetv2-lite and hitting same error. I feel doing similar thing as you, and want to know how did you fix the first problem.

Just applying onnx-simplifier?

[UPDATE] Hi again @dusty-nv,

I solved the initial issue, however,it seems that there is an incompability between onnx and tensorrt. I used your pytorch-detection repo to train and convert the model to ONNX. I obtained this error:

My tensorrt version is 5.1.6.

Could you help me on that? Thanks!

[NEW UPDATE]

After applying onnx-simplifier(https://github.com/daquexian/onnx-simplifier), I've made a little progress and then I got this error (this time related to Slice operation):

Thanks!

[NEW UPDATE2]

Ok, I found the problem! It is related to the conversion of regressional location results into boxes in the form of (center_x, center_y, h, w). Concretely, in:

if self.is_test:
            confidences = F.softmax(confidences, dim=2)
            boxes = convert_locations_to_boxes(
                locations, self.priors, self.config.center_variance, self.config.size_variance
            )
            boxes = center_form_to_corner_form(boxes)
            return confidences, boxes

Because, inside of the respective function, there is a:

torch.cat([
        locations[..., :2] * center_variance * priors[..., 2:] + priors[..., :2],
        torch.exp(locations[..., 2:] * size_variance) * priors[..., 2:]
    ], dim=locations.dim() - 1)

If you convert the trained model into ONNX in training mode (i.e. "is_test=False"), there is no error when running the detectnet-camera application. Thus, I'll try to fix this!

[j-beniwal]

Hi @dusty-nv ,

am also trying to load my custom model for inferencing using the inference libraries and codes from the repo. The following are some of the inputs for my issue.

1. am using the onnx converted ssdlinte model with name "ssdlite.onnx", laying in the same directory as the code.
2. want to inference on a video named "input.mp4", and want output as "output.mp4" laying in the same directory where the code is.

I tried using the commands as below: $python3 imagenet.py 3.mp4 output.mp4 --model=ssdlite.onnx --input_blob=input_0 --output_cvg=scores

the output image is attached

@rsamvelyan you have also worked out this issue can you also provide your inputs on this.