search

NVlabs / Deep_Object_Pose

Deep Object Pose Estimation (DOPE) – ROS inference (CoRL 2018)
Other
1.03k stars 287 forks source link

cv2.solvePnP failed with an error #265

Open utsavrai opened 2 years ago

utsavrai commented 2 years ago

I have generated 110k images of a custom object using 550 different HDRI maps. This was then trained with batch size of 20 and 50 epochs. The final minimum loss achieved was 0.00635. I have used the weights to test on generated dateset and actual camera images but I am getting following error as shown below. Screenshot from 2022-08-16 19-09-20 Screenshot from 2022-08-16 19-09-28

roslaunch dope dope.launch 
... logging to /home/ap/.ros/log/a35b2dc8-1d68-11ed-a1ac-239cc6e67a8b/roslaunch-01hw1365507-8604.log
Checking log directory for disk usage. This may take a while.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://01hw1365507:35595/

SUMMARY
========
CLEAR PARAMETERS
 * /dope/

PARAMETERS
 * /dope/class_ids/tray: 1
 * /dope/dimensions/tray: [2.2, 10.9, 23.7]
 * /dope/downscale_height: 400
 * /dope/draw_colors/tray: [255, 101, 0]
 * /dope/input_is_rectified: True
 * /dope/mesh_scales/tray: 0.01
 * /dope/meshes/tray: file:///home/ap/t...
 * /dope/overlay_belief_images: True
 * /dope/sigma: 3
 * /dope/thresh_angle: 0.5
 * /dope/thresh_map: 0.01
 * /dope/thresh_points: 0.1
 * /dope/topic_camera: /camera/color/ima...
 * /dope/topic_camera_info: /camera/color/cam...
 * /dope/topic_publishing: dope
 * /dope/weights/tray: package://dope/we...
 * /rosdistro: noetic
 * /rosversion: 1.15.14

NODES
  /
    dope (dope/dope)

ROS_MASTER_URI=http://localhost:11311

process[dope-1]: started with pid [8634]
Loading DOPE model '/home/ap/teleop_ws/src/Deep_Object_Pose/weights/tray.pth'...
    Model loaded in 4.756685972213745 seconds.
Running DOPE...  (Listening to camera topic: '/camera/color/image_raw')
Ctrl-C to stop
/home/ap/.local/lib/python3.8/site-packages/numpy/lib/function_base.py:959: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
  return array(a, order=order, subok=subok, copy=True)
7 valid points found
cv2.solvePnP failed with an error
5 valid points found
5 valid points found
7 valid points found
cv2.solvePnP failed with an error
8 valid points found
cv2.solvePnP failed with an error
8 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
6 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
5 valid points found
9 valid points found
cv2.solvePnP failed with an error
8 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
8 valid points found
cv2.solvePnP failed with an error
8 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
5 valid points found
5 valid points found
5 valid points found
9 valid points found
cv2.solvePnP failed with an error
5 valid points found
7 valid points found
cv2.solvePnP failed with an error
8 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
8 valid points found
cv2.solvePnP failed with an error
8 valid points found
cv2.solvePnP failed with an error
5 valid points found
5 valid points found
7 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
5 valid points found
9 valid points found
cv2.solvePnP failed with an error
8 valid points found
cv2.solvePnP failed with an error
6 valid points found
cv2.solvePnP failed with an error
5 valid points found
9 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
7 valid points found
cv2.solvePnP failed with an error
8 valid points found
cv2.solvePnP failed with an error
8 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
5 valid points found
9 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
5 valid points found
8 valid points found
cv2.solvePnP failed with an error
7 valid points found
cv2.solvePnP failed with an error
4 valid points found
9 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
8 valid points found
cv2.solvePnP failed with an error
8 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
8 valid points found
cv2.solvePnP failed with an error
8 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
8 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
7 valid points found
cv2.solvePnP failed with an error
6 valid points found
cv2.solvePnP failed with an error
5 valid points found
9 valid points found
cv2.solvePnP failed with an error
5 valid points found
1 valid points found
8 valid points found
cv2.solvePnP failed with an error
7 valid points found
cv2.solvePnP failed with an error
8 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
5 valid points found
7 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
6 valid points found
cv2.solvePnP failed with an error
5 valid points found
5 valid points found
9 valid points found
cv2.solvePnP failed with an error
6 valid points found
cv2.solvePnP failed with an error
3 valid points found
9 valid points found
cv2.solvePnP failed with an error
6 valid points found
cv2.solvePnP failed with an error
5 valid points found
9 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
5 valid points found
7 valid points found
cv2.solvePnP failed with an error
4 valid points found
9 valid points found
cv2.solvePnP failed with an error
8 valid points found
cv2.solvePnP failed with an error
6 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
5 valid points found
9 valid points found
cv2.solvePnP failed with an error
5 valid points found
5 valid points found
9 valid points found
cv2.solvePnP failed with an error
3 valid points found
8 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
7 valid points found
cv2.solvePnP failed with an error
9 valid points found
cv2.solvePnP failed with an error
8 valid points found
cv2.solvePnP failed with an error
6 valid points found

following is the config file

topic_camera: "/camera/color/image_raw"
topic_camera_info: "/camera/color/camera_info"
topic_publishing: "dope"
input_is_rectified: True   # Whether the input image is rectified (strongly suggested!)
downscale_height: 400      # if the input image is larger than this, scale it down to this pixel height

# Comment any of these lines to prevent detection / pose estimation of that object
weights: {
    "tray":"package://dope/weights/tray.pth",

}

# Cuboid dimension in cm x,y,z
dimensions: {

    "tray" : [2.2,10.9,23.7]
}

class_ids: {

    "tray"    : 1 
}

draw_colors: {

    "tray": [255, 101, 0]  # orange
}

# optional: provide a transform that is applied to the pose returned by DOPE
model_transforms: {
#    "cracker": [[ 0,  0,  1,  0],
#                [ 0, -1,  0,  0],
#                [ 1,  0,  0,  0],
#                [ 0,  0,  0,  1]]
}

# optional: if you provide a mesh of the object here, a mesh marker will be
# published for visualization in RViz
# You can use the nvdu_ycb tool to download the meshes: https://github.com/NVIDIA/Dataset_Utilities#nvdu_ycb
meshes: {

    "tray": "file:///home/ap/teleop_ws/src/Deep_Object_Pose/scripts/nvisii_data_gen/models/Tray/google_16k/tray.STL"
}

# optional: If the specified meshes are not in meters, provide a scale here (e.g. if the mesh is in centimeters, scale should be 0.01). default scale: 1.0.
mesh_scales: {

    "tray":    0.01,
}

overlay_belief_images: True   # Whether to overlay the input image on the belief images published on /dope/belief_[obj_name]

# Config params for DOPE
thresh_angle: 0.5
thresh_map: 0.01
sigma: 3
thresh_points: 0.1  # original
# thresh_points: 0.01

also find the camera info file ( this is the virtual camera used for generating dataset images, since we are testing on the dataset itself I have created the camera_info file with exact intrinsic values as the pybullet's virtual camera)

image_width: 500
image_height: 500
camera_name: insample
camera_matrix:
  rows: 3
  cols: 3
  data: [603.5535278320312, 0.0, 250.0, 0.0, 603.5535278320312, 250, 0.0, 0.0, 1.0]
distortion_model: plumb_bob
distortion_coefficients:
  rows: 1
  cols: 5
  data: [0, 0, 0, 0, 0]
rectification_matrix:
  rows: 3
  cols: 3
  data: [1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0]
projection_matrix:
  rows: 3
  cols: 4
  data: [603.5535278320312, 0.0, 250.0, 0.0, 0.0, 603.5535278320312, 250, 0.0, 0.0, 0.0, 1.0, 0.0]
TontonTremblay commented 2 years ago

Depending on the version of opencv, there might be a problem with the solvepnp.

https://github.com/NVlabs/Deep_Object_Pose/blob/master/src/dope/inference/cuboid_pnp_solver.py#L77-L95

Can you try different pnp versions? Let me know if that helps you out. Sorry about that.