search

dusty-nv / ros_deep_learning

Deep learning inference nodes for ROS / ROS2 with support for NVIDIA Jetson and TensorRT
887 stars 258 forks source link

segnet - map classes in source image with class_mask #63

Open murikill opened 3 years ago

murikill commented 3 years ago

I'm trying to get the class of each pixel in the source image.

I'm using segnet node and it works well, I get on the output side colored image and grayscale color mask.

Next I upscaled the class mask to the source image size and colored the pixel into the class color. But the output doesn't make sense to me (the muster of color and mask image are not equal...) Is there some step more, that I have to do?

Tank you

segnet_output

dusty-nv commented 3 years ago

Hi @murikill , internally the class_mask is used to generate the color_mask (with bilinear upsampling applied).

It looks like when you resized the class_mask, it was filled with mostly zeros, so I think the resizing did not work properly.

However, you can have the API resize the class_mask for you to your desired size - see here:

https://github.com/dusty-nv/ros_deep_learning/blob/ac40e93413f4b7cb911a18c0e4d5daac479234d4/src/node_segnet.cpp#L160

Just change that to publish_mask_class(input->width, input->height)

Note that I recommend keeping the class mask at it's original size (net->GetGridWidth(), net->GetGridHeight()), because it is not providing more information by upsampling it - while only increasing the processing requirements.

murikill commented 3 years ago

Hi @dusty-nv, thank you for your reply.

Indeed the resizing stuff was faulty. There was not enough arguments in the opencv method, here is the line correct to upsampling the mask : cv::resize(image_input, image_resized, cv::Size(width, height),0, 0, cv::INTER_NEAREST);

In the next picture I changed the upsampling method of color_mask to point. The output lines in the terminal are the values of raw class_mask image (32x16 px), I did nothing with it. On the bottom right image (image_debug), I resized and colored the class_mask, just to visualize the results.

segnet_output2

I've tryed to change the size of mask in the API directly, and I got followed output:

segnet_output3

It looks a bit better, but the bottom half is filled with zero values and there are times the same image in a upper image half.

In compare to origin resolution (image bottom) of mask (32*16 px), color_mask and class_mask looks very similar.

segnet_output4

color_mask and class_mask are raw images published from segnet node.

I didn't understand the output of the class_mask :( You are creating color_mask with the same class_mask data, or not? It's possible that the class_mask will not correct converted to ros message?

Is there any possible to get the probability of the classes?

BogdanLazarescu commented 3 years ago

I am facing the same issue. Did you find a solution to it?

murikill commented 3 years ago

I don't. I'm using color_mask to get the class reference to each pixel in the original image. It isn't fine, but as workaround it works.

BogdanLazarescu commented 3 years ago

I found a quick fix as well. The issue come from the Convert method in image_converter.cpp. You can patch it by using something like: if (format != IMAGE_GRAY8) memcpy(msg.data.data(), mInputCPU, msg_size); else memcpy(msg.data.data(), mOutputCPU, msg_size); ~line 181

jodusan commented 2 years ago

@dusty-nv Hey, can you check this out? https://github.com/dusty-nv/ros_deep_learning/pull/100

dusty-nv commented 2 years ago

OK thanks - just merged this.