Inference speed enhancement

[vijaysamula]

Hi, I am glad about your work. I am using DenseFusion for real-time implementation(ROS). I have a problem with the speed of detection (inference speed) is very high (1 frame per 9sec). How to resolve it?

[j96w]

Hi, please double check your computer settings and the way of using this code. As is reported in the paper, we achieved 16 fps tracking speed with a 1080Ti Nvidia GPU in the real robot experiment. Also in another more complex category-level tacking project, DenseFusion also achieves 10 fps inference speed.

[vijaysamula]

I didn't do much in ROS. Just created a node that subscribes to the data from RGB, Depth, and label. and plotting the results on RGB using Matplotlib.

[j96w]

Hi, could you list your computer hardware settings? I thought you were talking about the inference speed.

[vijaysamula]

The PC is with RTX 2080 TI, i7 8th gen, and RAM 16gb

[j96w]

Ok, so you mean that your computer with RTX2080 takes 9 seconds to get the result of one single frame? Sorry, but I think there must be some bugs.

[vijaysamula]

can you check my code once?

[vijaysamula]

I will mail you if you provide me mail id. if it is possible only.

[an99990]

hi @vijaysamula, how did you manage to change the code for inference ? I am facing issues understanding the arguments the models needs img if the rgb image, points is the point cloud , but what is choose and idx ? estimator(img, points, choose, idx)

thank you

[vijaysamula]

Hi @an99990, The choose are the points [choose] (https://github.com/j96w/DenseFusion/blob/f7af2d99a38890183fe1c97d84f6bf1488a175f5/tools/eval_ycb.py#L156) and index corresponds to the object index in the image. Densefusion first segments and crops the object and finds the corresponding depth mask and estimates the pose.

Coming to inference, images can be subscribed and points are always fixed. The index comes from the segmented pixel value. Just modify it accordingly.

[an99990]

thank you for your answer @vijaysamula , So right now i am working with a zed camera, i am subscribed to an image topic and the point cloud topic, i feed these for my first models which outputs segmentation mask of the same size as the image.

From what you explain index would be the class label ( 1,2,3 ) of the object in the mask ? I would need to crop the object from the masks, and feed that mask ROI to densefusion ? Also i saw the definition of choose but i dont understand its purpose and values, and you say points are always fixed ? the point cloud are fixed ? what do you mean

Thank you again for your answer,

[vijaysamula]

The point cloud can be subscribed to and used. But I used the point cloud given by the YCB dataset as input without real-time. The RGB image is used to detect objects and choose is the variable that extracts the corresponding object https://github.com/j96w/DenseFusion/blob/f7af2d99a38890183fe1c97d84f6bf1488a175f5/tools/eval_ycb.py#L156.

[an99990]

and object index in the image means that if theres one object in the image, index would be 1 ? in real time scenario is that something we give to the model ourselves such as the index is the list of mask for example if len(mask) = nb_ instance, then index would iterate in that range ?

thank you @vijaysamula

[an99990]

what about cam_R_m2c and cam_t_m2c how are these values supposed to be feed in real time purposes ?

[vijaysamula]

The index comes from segmented pixel value as it is the object correspondence. I don't know about cam_R_m2c and cam_t_m2c? Point the line here.

[an99990]

from dataset.py in linemod dataset

https://github.com/j96w/DenseFusion/blob/f7af2d99a38890183fe1c97d84f6bf1488a175f5/datasets/linemod/dataset.py#L130 https://github.com/j96w/DenseFusion/blob/f7af2d99a38890183fe1c97d84f6bf1488a175f5/datasets/linemod/dataset.py#L131

[vijaysamula]

I used it for YCB. I don't know about Linemod dataset.

[an99990]

ok thank you, i will try with YCB then

[an99990]

@vijaysamula do you know what num_points should be, i see it always being 500, is num_objects the number of classes ? thank you