Failed detected

[QuanNguyen94]

I load cracker_60.pth checkpoint and try the below image, but have no cube detected. The result list is empty, and I have tried showing belief maps and affinity maps but they look fail. Do you have any ideas about this, can you explain more about affinity map pls ?

[TontonTremblay]

I would say that the problem comes from the texture on the cracker box not being the same as the one we are using to train our model. Can you try to point the camera to your monitor with an image of the actual cracker box we used? Also the belief maps look pretty bad, I would think the top vertexes should react to the label.

The affinity maps are used to find multiple instances of the same object. Technically for this release they are not essential assuming there is only one object of that type in the scene.

[TontonTremblay]

try this one: https://www.google.com/search?q=cheez+it+original&safe=active&rlz=1C1GGRV_enUS764US765&source=lnms&tbm=isch&sa=X&ved=0ahUKEwiiyf6WltXeAhVcIjQIHW-RAeIQ_AUIDygC&biw=1745&bih=917#imgrc=sR9i3ISjk3rFGM:

[QuanNguyen94]

I have try this one, but got the same issue. Can you give me a specific image you used ?

[TontonTremblay]

Here a few images that we use in the paper: https://www.dropbox.com/s/uo0nuhu9g5i1yj0/baxter.tar?dl=0

What is the resolution of images that you are sending to the network? The height has to be between 400 and 500.

Have you try different weights with different objects?

[QuanNguyen94]

Thank for your sharing, I have tried it. Is this result looks good ? Some more question pls, i wonder how you label for affinity maps and is there any special things in loss function, or just use L2 loss of BeliefMap and AffinityMap with groundtruth ?

[yuzw213]

Thank for your sharing, I have tried it. Is this result looks good ? Some more question pls, i wonder how you label for affinity maps and is there any special things in loss function, or just use L2 loss of BeliefMap and AffinityMap with groundtruth ?

Hello!

How do you achieve the effect of dope in your own camera? I compiled it smoothly, but I can't get the image, or I can't achieve the desired effect in the ROS camera. Can you help me? Thank you!

[QuanNguyen94]

Hello bro, I didnt setup the camera yet so I cannot say anything about your question.

[TontonTremblay]

What I am seeing is correct, this is the expected behaviour. We simply use L2 on the maps, you are correct. There is no special loss.

[QuanNguyen94]

Thank you @TontonTremblay, that 's all. You and your team had a great work :D

[Abdul-Mukit]

Hi guys, here is my code for visualizing the feature maps. The detector is provided by DOPE. Hope it helps.

import cv2
import matplotlib.pyplot as plt
import torch
from detector import *

## Settings
name = 'mustard'
net_path = 'data/net/mustard_60.pth'
# net_path = 'data/net/cautery_s6_60.pth'
gpu_id = 0
img_path = 'data/images/cautery_DR_2.png'
# img_path = 'data/images/cautery_real_1.jpg'

# Function for visualizing feature maps
def viz_layer(layer, n_filters=9):
    fig = plt.figure(figsize=(20, 20))
    row = 1
    for i in range(n_filters):
        ax = fig.add_subplot(4, 5, i + 1, xticks=[], yticks=[])
        # grab layer outputs
        ax.imshow(np.squeeze(layer[i].data.numpy()), cmap='gray')
        ax.set_title('Output %s' % str(i + 1))

# load color image
in_img = cv2.imread(img_path)
# in_img = cv2.resize(in_img, (640, 480))
in_img = cv2.cvtColor(in_img, cv2.COLOR_BGR2RGB)
# plot image
plt.imshow(in_img)

model = ModelData(name, net_path, gpu_id)
model.load_net_model()
net_model = model.net

# Run network inference
image_tensor = transform(in_img)
image_torch = Variable(image_tensor).cuda().unsqueeze(0)
out, seg = net_model(image_torch)
vertex2 = out[-1][0].cpu()
aff = seg[-1][0].cpu()

# View the vertex and affinities
viz_layer(vertex2)
viz_layer(aff, n_filters=16)

plt.show()

[Abduoit]

Thanks @Abdul-Mukit , do you know how to print the output of the vertices (position of vertices), to show the centroids of the object.

[zhanghui-hunan]

@Abdul-Mukit thanks for your great job! I have seen your recent work posted to youtube which detected pen. Thus, I am eager to know how do you create the pen model and get the dataset by NDDS? Hope for your reply, sincerely! PS: as far as I know, can MAYA and substance painter get the model, but not for sure.

[Abdul-Mukit]

Thanks @Abdul-Mukit , do you know how to print the output of the vertices (position of vertices), to show the centroids of the object.

Sorry, didn't try that. I think you can find the required code if you look into the original codes.

[Abdul-Mukit]

@Abdul-Mukit thanks for your great job! I have seen your recent work posted to youtube which detected pen. Thus, I am eager to know how do you create the pen model and get the dataset by NDDS? Hope for your reply, sincerely! PS: as far as I know, can MAYA and substance painter get the model, but not for sure.

@liangzhicong456 Thanks for checking out the videos. The model was created in blender. But you can use MAYA too, I think. Then you have to import just the mesh as fbx (nothing else worked for me). Then in UE4, you add different materials to different parts of your 3d model. Hope that helps. BTW just out of curiosity, what object are you trying for?

[zhanghui-hunan]

@Abdul-Mukit Thanks for your reply, I want to create the tissue model, but the complex surface texture makes me unable to start which the texture information is very important for detection. So I really want to know how I can get more accurate model and texture information. I have known Blender before, but can get accurate model? I want to know that all the information inside is hand-painted, including models and textures? I am a beginner and may have some stupid questions, hope for your patient answer. Sample tissue image is below:

[Abdul-Mukit]

@liangzhicong456 I am a beginner myself, It's just been 5 months since I started my project :) From the image, it looks to me hand-designing the texture is too difficult. You need UV maps for texture. If I were you, I guess I would try to load the YCB objects (the objects used in DOPE) in NDDS. You will need to learn how to load UV maps along with mesh I think. You will need to do something similar for your tissue box. If you don't want to go for a 3D scan then you can try to create a UV map or a texture map by slicing the packaging of the tissue box then taking pictures of it. I think it should be possible. I saw some tutorials about UV mapping and it appears to me UV maps will solve your problem.

[zhanghui-hunan]

@Abdul-Mukit Thanks for your suggestion. I found that Blender combined with UV map can get the effect I want. I am studying hard now, hope for keeping communication with you, thanks sincerely！

[Abdul-Mukit]

@liangzhicong456 you can email me in my email address (in my profile) instead of lengthening this issue.

[zhanghui-hunan]

@Abdul-Mukit ok，thank you very much for your help！

[aditya2592]

Hi, I am trying to detect the pepsi can in below image. After training for 60 epochs on a dataset created in NDDS, I am able to see activation in the belief and affinity maps (using the code provided by Abdul above). But when I run the detector through the dope node, I dont get bounding boxes for the object. What could be the reason for this? Is it happening because the network is getting confused between sprite and pepsi can?

[TontonTremblay]

Which object are you hoping to detect?

[aditya2592]

My goal is to be able to detect all 6 objects in the below image.

Currently I am trying with pepsi_can (the last on to the right in the image). I was able to get some results. My mistake was not converting BGR to RGB after reading image using opencv. However looking at the results, the depth from the camera looks significantly off. Will training on more images with random backgrounds help? Currently I am training with the same background because my application assumes that the background is going to be same between training and test

[TontonTremblay]

Are you trying to run in the real world? Or you are running inference on the training data you generated?

[aditya2592]

I am running inference on the training data I generated. Not planning to run in the real world as of now.

[TontonTremblay]

The output you are getting is similar to what I would think you should get. The pepsi can you have has some almost symmetries, which explains why output 8 is kind of multi-modal. I dont think you can avoid the confusion with the sprite can as the colors and top are very similar. The way you visualize the affinity field is just miss leading, you should display them as color wheels.

The color shows the direction of the vector and you will have to ignore everything that is close to zero in magnitude.

[aditya2592]

Thanks for the suggestion @TontonTremblay. After some tweaks I am getting the 3D bounding box correctly as below : However, the depth (z coordinate in camera frame) was off by 10-12 cm. That was fixed when I changed the downscale height to 400 (the height the network was trained on). Going to try the other objects now.

[TontonTremblay]

This is awesome, I am glad you were able to get good results :P. Yeah you have to be careful with image size when running the algorithm.

[jediofgever]

Hi, I have exactly same problem. the activation maps seems to be activated but the dope node detects nothing on the image. I tried to convert bgr to rgb when reading image through opencv, resized the image but still the object is not being detected. What else could be problem ?

[TontonTremblay]

Lets try to run DOPE with a lower threshold.

https://github.com/NVlabs/Deep_Object_Pose/blob/1ee8e0ddaa01886785ba176581582947008440d4/src/dope/inference/detector.py#L314

https://github.com/NVlabs/Deep_Object_Pose/blob/1ee8e0ddaa01886785ba176581582947008440d4/src/dope/inference/detector.py#L366

You can change those values in the config file. Play around with some values and let me know what you are getting.

I think I am going to provide a debug view for the affinity maps. I cannot read the values you are displaying.

[akeaveny]

Hi @Abdul-Mukit, I am attempting to visualizing the feature maps but I am unable to run your script as a stand alone file. The error I can't get past is

Traceback (most recent call last): File "./visualize.py", line 7, in <module> from detector import * ModuleNotFoundError: No module named 'detector'

[Abdul-Mukit]

@akeaveny you need to put detector.py in the same directory as your code. You can find the detector.py file in DOPE.

[akeaveny]

@Abdul-Mukit ahh, thanks! Got it running now

[akeaveny]

@jediofgever were you able to resolve your issue? I'm facing the same problem

[Abdul-Mukit]

@QuanNguyen94 hi again. I am trying to generate the affinity map visualization using color wheels as you commented above. link. Could you please tell me how you generated those plots?