search

NVlabs / Deep_Object_Pose

Deep Object Pose Estimation (DOPE) – ROS inference (CoRL 2018)
Other
1.02k stars 287 forks source link

Error when using with ZED camera. ERROR: Unable to open camera for capture. Is camera plugged in? #106

Open astronaut71 opened 4 years ago

astronaut71 commented 4 years ago

Hello

Im using ZED camera with Ubuntu 16.04, CUDA 10 and cudnn 7.1. I change the config file according the publishing ros topics like this

topic_camera: "/zed/zed_node/rgb/image_rect_color"
topic_camera_info: "/zed/zed_node/rgb_raw/camera_info"
topic_publishing: "dope_zed"
input_is_rectified: False   # Whether the input image is rectified (strongly suggested!)
downscale_height: 400      # if the input image is larger than this, scale it down to this pixel height

Then the camera info like this


image_width: 640
image_height: 480
camera_name: dope/webcam
camera_matrix:
  rows: 3
  cols: 3
  data: [679.4, 0, 621.9, 0, 679.4, 349.6, 0, 0, 1]
distortion_model: plumb_bob
distortion_coefficients:
  rows: 1
  cols: 5
  data: [0, 0, 0, 0, 0]
rectification_matrix:
  rows: 3
  cols: 3
  data: [1, 0, 0, 0, 1, 0, 0, 0, 1]
projection_matrix:
  rows: 3
  cols: 4
  data: [679.4, 0, 621.9, 0, 0, 679.4, 349.6, 0, 0, 0, 1, 0]

When run the ZED launch with` roslaunch zed_wrapper zed.launch`in one terminal and in the other the DOPE with `roslaunch dope camera.launch` I got the following error

> [INFO] [1586605950.736040]: camera calibration URL: package://dope/config/camera_info.yaml
> /opt/ros/kinetic/lib/python2.7/dist-packages/camera_info_manager/camera_info_manager.py:453: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
>   calib = yaml.load(f)
> [WARN] [1586605950.792829]: [dope_webcam_-1] does not match name dope/webcam in file /home/admini/catkin_deep_object_pose/src/dope/config/camera_info.yaml
> VIDEOIO ERROR: V4L2: Pixel format of incoming image is unsupported by OpenCV
> Unable to stop the stream: Device or resource busy
> [FATAL] [1586605950.793150]: ERROR:  Unable to open camera for capture.  Is camera plugged in?
> [dope_webcam-1] process has died [pid 22148, exit code 1, cmd /home/admini/catkin_deep_object_pose/src/dope/nodes/camera __name:=dope_webcam __log:=/home/admini/.ros/log/d104a0d0-7be7-11ea-8110-309c23881ba5/dope_webcam-1.log].
> log file: /home/admini/.ros/log/d104a0d0-7be7-11ea-8110-309c23881ba5/dope_webcam-1*.log
> all processes on machine have died, roslaunch will exit
> shutting down processing monitor...
> ... shutting down processing monitor complete

Please any help?
TontonTremblay commented 4 years ago

What you should do is write a manual config file for the intrinsic for the camera and publish that as a topic for dope to use, I think @mintar wrote a nice tutorial on how to do that.

astronaut71 commented 4 years ago

sorry dont understand all what you saying. The camera_info.yaml file is with intrinsic parameters of the ZED camera. Where is the @mintar tutorial? Sorry I come to another error when $ roslaunch dope camera.launch and $ roslaunch dope dope.launch

Please can you look at that issue I posted?

mintar commented 4 years ago

When run the ZED launch withroslaunch zed_wrapper zed.launchin one terminal and in the other the DOPE with roslaunch dope camera.launch I got the following error

Yes, you shouldn't do that. You should run the ZED camera driver in one terminal and DOPE (roslaunch dope dope.launch) in the other. Don't start roslaunch dope camera.launch at all.

astronaut71 commented 4 years ago

I have done that what @mintar said and then load the model

Loading DOPE model '/home/admini/catkin_deep_object_pose/src/dope/weights/cracker_60.pth'... /home/admini/.local/lib/python2.7/site-packages/torch/cuda/init.py:114: UserWarning: Found GPU0 GeForce RTX 2080 Ti which requires CUDA_VERSION >= 9000 for optimal performance and fast startup time, but your PyTorch was compiled with CUDA_VERSION 8000. Please install the correct PyTorch binary using instructions from http://pytorch.org

warnings.warn(incorrect_binary_warn % (d, name, 9000, CUDA_VERSION)) Model loaded in 299.177142143 seconds.

Then come the ERROR:

[ERROR] [1586751625.186371]: bad callback: <bound method Subscriber.callback of <message_filters.Subscriber object at 0x7f7381ca7750>> Traceback (most recent call last): File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/topics.py", line 750, in _invoke_callback cb(msg) File "/opt/ros/kinetic/lib/python2.7/dist-packages/message_filters/init.py", line 75, in callback self.signalMessage(msg) File "/opt/ros/kinetic/lib/python2.7/dist-packages/message_filters/init.py", line 57, in signalMessage cb((msg + args)) File "/opt/ros/kinetic/lib/python2.7/dist-packages/message_filters/init.py", line 224, in add self.signalMessage(msgs) File "/opt/ros/kinetic/lib/python2.7/dist-packages/message_filters/init.py", line 57, in signalMessage cb((msg + args)) File "/home/admini/catkin_deep_object_pose/src/dope/nodes/dope", line 248, in image_callback self.config_detect File "/home/admini/catkin_deep_object_pose/src/dope/src/dope/inference/detector.py", line 251, in detect_object_in_image out, seg = net_model(image_torch) File "/home/admini/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in call result = self.forward(input, kwargs) File "/home/admini/.local/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 112, in forward return self.module(*inputs[0], *kwargs[0]) File "/home/admini/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in call result = self.forward(input, kwargs) File "/home/admini/catkin_deep_object_pose/src/dope/src/dope/inference/detector.py", line 91, in forward out1 = self.vgg(x) File "/home/admini/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in call result = self.forward(*input, *kwargs) File "/home/admini/.local/lib/python2.7/site-packages/torch/nn/modules/container.py", line 91, in forward input = module(input) File "/home/admini/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in call result = self.forward(input, **kwargs) File "/home/admini/.local/lib/python2.7/site-packages/torch/nn/modules/conv.py", line 301, in forward self.padding, self.dilation, self.groups)

Any help please?

astronaut71 commented 4 years ago

I change the pytorch with pip install torch==1.2.0 torchvision==0.4.0 -f https://download.pytorch.org/whl/torch_stable.html then the error disappear and load the model fast within 5 sec with

Model loaded in 4.65109205246 seconds. Running DOPE... (Listening to camera topic: '/zed/zed_node/rgb/image_rect_color')

The rostopic are:

/dope/detected_objects /dope/dimension_cracker /dope/markers /dope/pose_cracker /dope/rgb_points

Is missing the /dope/rgb/image_rect_color Why is not published? Them nothing happened. So what is the problem?

mintar commented 4 years ago

About your "bad callback" error : I think you cut out the most interesting part of the error message. It should say something like "incompatible types", not just bad callback. Also, please format your logs as code. Makes it much easier to read.

astronaut71 commented 4 years ago

ok sure. Should I reproduce the "bad callback" error again. As said when upgrade the pytorch that error is not anymore. But Now I dont have the image topic. Please can help?

astronaut71 commented 4 years ago

this are my camera_ino.yaml

image_width: 640
image_height: 480
camera_name: zed_-1
camera_matrix:
  rows: 3
  cols: 3
  data: [679.4, 0, 621.9, 0, 679.4, 349.6, 0, 0, 1]
distortion_model: plumb_bob
distortion_coefficients:
  rows: 1
  cols: 5
  data: [0, 0, 0, 0, 0]
rectification_matrix:
  rows: 3
  cols: 3
  data: [1, 0, 0, 0, 1, 0, 0, 0, 1]
projection_matrix:
  rows: 3
  cols: 4
  data: [679.4, 0, 621.9, 0, 0, 679.4, 349.6, 0, 0, 0, 1, 0]

then config_pose.yaml

topic_camera: "/zed/zed_node/rgb/image_rect_color"
topic_camera_info: "/zed/zed_node/rgb_raw/camera_info"
topic_publishing: "dope"
input_is_rectified: True   # Whether the input image is rectified (strongly suggested!)
downscale_height: 500      # if the input image is larger than this, scale it down to this pixel height

# Comment any of these lines to prevent detection / pose estimation of that object
weights: {
    "cracker":"package://dope/weights/cracker_60.pth",
    # "gelatin":"package://dope/weights/gelatin_60.pth",
    # "meat":"package://dope/weights/meat_20.pth",
    # "mustard":"package://dope/weights/mustard_60.pth",
    # "soup":"package://dope/weights/soup_60.pth",
    #"sugar":"package://dope/weights/sugar_60.pth",
    # "bleach":"package://dope/weights/bleach_28_dr.pth"
}

# Cuboid dimension in cm x,y,z
dimensions: {
    "cracker": [16.403600692749023,21.343700408935547,7.179999828338623],
    "gelatin": [8.918299674987793, 7.311500072479248, 2.9983000755310059],
    "meat": [10.164673805236816,8.3542995452880859,5.7600898742675781],
    "mustard": [9.6024150848388672,19.130100250244141,5.824894905090332],
    "soup": [6.7659378051757813,10.185500144958496,6.771425724029541],
    "sugar": [9.267730712890625,17.625339508056641,4.5134143829345703],
    "bleach": [10.267730712890625,26.625339508056641,7.5134143829345703],
}

class_ids: {
    "cracker": 1,
    "gelatin": 2,
    "meat":    3,
    "mustard": 4,
    "soup":    5,
    "sugar":   6,
    "bleach":  7,
}

draw_colors: {
    "cracker": [13, 255, 128],  # green
    "gelatin": [255, 255, 255],  # while
    "meat": [0, 104, 255],  # blue
    "mustard": [217,12, 232],  # magenta
    "soup": [255, 101, 0],  # orange
    "sugar": [232, 222, 12],  # yellow
    "bleach": [232, 222, 12],  # yellow
}

# optional: provide a transform that is applied to the pose returned by DOPE
model_transforms: {
#    "cracker": [[ 0,  0,  1,  0],
#                [ 0, -1,  0,  0],
#                [ 1,  0,  0,  0],
#                [ 0,  0,  0,  1]]
}

# optional: if you provide a mesh of the object here, a mesh marker will be
# published for visualization in RViz
# You can use the nvdu_ycb tool to download the meshes: https://github.com/NVIDIA/Dataset_Utilities#nvdu_ycb
meshes: {
#    "cracker": "file://path/to/Dataset_Utilities/nvdu/data/ycb/aligned_cm/003_cracker_box/google_16k/textured.obj",
#    "gelatin": "file://path/to/Dataset_Utilities/nvdu/data/ycb/aligned_cm/009_gelatin_box/google_16k/textured.obj",
#    "meat":    "file://path/to/Dataset_Utilities/nvdu/data/ycb/aligned_cm/010_potted_meat_can/google_16k/textured.obj",
#    "mustard": "file://path/to/Dataset_Utilities/nvdu/data/ycb/aligned_cm/006_mustard_bottle/google_16k/textured.obj",
#    "soup":    "file://path/to/Dataset_Utilities/nvdu/data/ycb/aligned_cm/005_tomato_soup_can/google_16k/textured.obj",
#    "sugar":   "file://path/to/Dataset_Utilities/nvdu/data/ycb/aligned_cm/004_sugar_box/google_16k/textured.obj",
#    "bleach":  "file://path/to/Dataset_Utilities/nvdu/data/ycb/aligned_cm/021_bleach_cleanser/google_16k/textured.obj",
}

# optional: If the specified meshes are not in meters, provide a scale here (e.g. if the mesh is in centimeters, scale should be 0.01). default scale: 1.0.
mesh_scales: {
    "cracker": 0.01,
    "gelatin": 0.01,
    "meat":    0.01,
    "mustard": 0.01,
    "soup":    0.01,
    "sugar":   0.01,
    "bleach":  0.01,
}

# Config params for DOPE
thresh_angle: 0.5
thresh_map: 0.01
sigma: 3
thresh_points: 0.1

then dope.launch is same and my zed.launch is

<launch>
    <arg name="svo_file"             default="" /> <!-- <arg name="svo_file" default="path/to/svo/file.svo"> -->
    <arg name="stream"               default="" /> <!-- <arg name="stream" default="<ip_address>:<port>"> -->

    <arg name="node_name"            default="zed_node" />
    <arg name="camera_model"         default="zed" /> <!-- 'zed' or 'zedm' -->
    <arg name="publish_urdf"         default="true" />

 <node name="dope_webcam" pkg="dope" type="zed" output="screen">
    <param name="camera_info_url" value="package://dope/config/camera_info.yaml" />
  </node>
    <group ns="zed">
        <include file="$(find zed_wrapper)/launch/zed_camera.launch">
            <arg name="svo_file"            value="$(arg svo_file)" />
            <arg name="stream"              value="$(arg stream)" />
            <arg name="node_name"           value="$(arg node_name)" />
            <arg name="camera_model"        value="$(arg camera_model)" />
            <arg name="publish_urdf"        value="$(arg publish_urdf)" />
        </include>
    </group>
</launch>

Im not sure about write a manual config file for the intrinsic for the camera and publish that as a topic for dope to use. I saw your tutorial bit still not clear as ZED already is calibrated and I just put the instrisic parameter in the dope config.

Also when rostopic info /zed/zed_node/rgb_raw/camera_info I got 

Type: sensor_msgs/CameraInfo

Publishers: 
 * /zed/zed_node (http://msi370:37807/)

Subscribers: 
 * /dope (http://msi370:36571/)

So means topic is published and dope subscribed. But still nothing happened. Please any help?

nourmorsy commented 8 months ago

can anyone help me i have the same error the first one however i uses astra camera @mintar ?

nourmorsy commented 8 months ago

when i run camera.launch i get this error [FATAL] [1709456760.896459]: unable to register service [/{'depth': {'image_rect_raw': {'compressedDepth': {'depth_max': 10.0, 'png_level': 1, 'depth_quantization': 100.0}, 'compressed': {'png_level': 9, 'jpeg_quality': 80, 'format': 'jpeg'}, 'theora': {'optimize_for': 1, 'target_bitrate': 800000, 'quality': 31, 'keyframe_frequency': 64}}, 'image': {'compressedDepth': {'depth_max': 10.0, 'png_level': 1, 'depth_quantization': 100.0}, 'compressed': {'png_level': 9, 'jpeg_quality': 80, 'format': 'jpeg'}, 'theora': {'optimize_for': 1, 'target_bitrate': 800000, 'quality': 31, 'keyframe_frequency': 64}}, 'image_rect': {'compressedDepth': {'depth_max': 10.0, 'png_level': 1, 'depth_quantization': 100.0}, 'compressed': {'png_level': 9, 'jpeg_quality': 80, 'format': 'jpeg'}, 'theora': {'optimize_for': 1, 'target_bitrate': 800000, 'quality': 31, 'keyframe_frequency': 64}}, 'image_raw': {'compressedDepth': {'depth_max': 10.0, 'png_level': 1, 'depth_quantization': 100.0}, 'compressed': {'png_level': 9, 'jpeg_quality': 80, 'format': 'jpeg'}, 'theora': {'optimize_for': 1, 'target_bitrate': 800000, 'quality': 31, 'keyframe_frequency': 64}}}, 'depth_registered_rectify_depth': {'interpolation': 0}, 'ir': {'image': {'compressedDepth': {'depth_max': 10.0, 'png_level': 1, 'depth_quantization': 100.0}, 'compressed': {'png_level': 9, 'jpeg_quality': 80, 'format': 'jpeg'}, 'theora': {'optimize_for': 1, 'target_bitrate': 800000, 'quality': 31, 'keyframe_frequency': 64}}}, 'driver': {'use_device_time': True, 'rgb_preferred': True, 'ir_mode': 5, 'color_mode': 5, 'z_offset_mm': 0, 'depth_time_offset': -0.033, 'depth_mode': 5, 'data_skip': 0, 'rgb_camera_info_url': '', 'auto_exposure': True, 'ir_time_offset': -0.033, 'bootorder': 0, 'depth_camera_info_url': '', 'depth_ir_offset_y': 4.0, 'depth_ir_offset_x': 5.0, 'z_scaling': 1.0, 'device_id': '2bc5:0401', 'devnums': 1, 'auto_white_balance': True, 'depth_frame_id': 'camera_depth_optical_frame', 'color_time_offset': -0.033, 'depth_registration': True, 'rgb_frame_id': 'camera_rgb_optical_frame', 'color_depth_synchronization': False}, 'camera_nodelet_manager': {'num_worker_threads': 4}, 'rgb_rectify_color': {'interpolation': 1}, 'depth_registered': {'image_raw': {'compressedDepth': {'depth_max': 10.0, 'png_level': 1, 'depth_quantization': 100.0}, 'compressed': {'png_level': 9, 'jpeg_quality': 80, 'format': 'jpeg'}, 'theora': {'optimize_for': 1, 'target_bitrate': 800000, 'quality': 31, 'keyframe_frequency': 64}}, 'image': {'compressedDepth': {'depth_max': 10.0, 'png_level': 1, 'depth_quantization': 100.0}, 'compressed': {'png_level': 9, 'jpeg_quality': 80, 'format': 'jpeg'}, 'theora': {'optimize_for': 1, 'target_bitrate': 800000, 'quality': 31, 'keyframe_frequency': 64}}, 'hw_registered': {'image_rect_raw': {'compressedDepth': {'depth_max': 10.0, 'png_level': 1, 'depth_quantization': 100.0}, 'compressed': {'png_level': 9, 'jpeg_quality': 80, 'format': 'jpeg'}, 'theora': {'optimize_for': 1, 'target_bitrate': 800000, 'quality': 31, 'keyframe_frequency': 64}}, 'image_rect': {'compressedDepth': {'depth_max': 10.0, 'png_level': 1, 'depth_quantization': 100.0}, 'compressed': {'png_level': 9, 'jpeg_quality': 80, 'format': 'jpeg'}, 'theora': {'optimize_for': 1, 'target_bitrate': 800000, 'quality': 31, 'keyframe_frequency': 64}}}}, 'rgb': {'image_raw': {'compressedDepth': {'depth_max': 10.0, 'png_level': 1, 'depth_quantization': 100.0}, 'compressed': {'png_level': 9, 'jpeg_quality': 80, 'format': 'jpeg'}, 'theora': {'optimize_for': 1, 'target_bitrate': 800000, 'quality': 31, 'keyframe_frequency': 64}}, 'image_rect_color': {'compressedDepth': {'depth_max': 10.0, 'png_level': 1, 'depth_quantization': 100.0}, 'compressed': {'png_level': 9, 'jpeg_quality': 80, 'format': 'jpeg'}, 'theora': {'optimize_for': 1, 'target_bitrate': 800000, 'quality': 31, 'keyframe_frequency': 64}}}, 'depth_rectify_depth': {'interpolation': 0}}set_camera_info] with master: ERROR: parameter [service] contains illegal chars [INFO] [1709456760.897583]: camera calibration URL: package://dope/config/camera_info.yaml /home/nour/catkin_ws/src/camera_info_manager_py/src/camera_info_manager/camera_info_manager.py:438: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. calib = yaml.load(f) [ WARN:0] global /tmp/pip-req-build-vu_aq9yd/opencv/modules/videoio/src/cap_v4l.cpp (893) open VIDEOIO(V4L2:/dev/video0): can't open camera by index [FATAL] [1709456760.921139]: ERROR: Unable to open camera for capture. Is camera plugged in? [dope_webcam-1] process has died [pid 3277, exit code 1, cmd /home/nour/catkin_ws/src/dope/nodes/camera __name:=dope_webcam __log:=/home/nour/.ros/log/1a0ecdbc-d93d-11ee-a16d-18c04d44fae3/dope_webcam-1.log]. log file: /home/nour/.ros/log/1a0ecdbc-d93d-11ee-a16d-18c04d44fae3/dope_webcam-1*.log all processes on machine have died, roslaunch will exit shutting down processing monitor... ... shutting down processing monitor complete done

nourmorsy commented 8 months ago

however when i run dope.launch it works do i need to change something ?

TontonTremblay commented 8 months ago 
with master: ERROR: parameter [service] contains illegal chars

I think you yaml is broken somehow.