upper_body_detector not working for ZR300

[Anitawuxy]

Hello everyone, i am using the intel realsense camera ZR300, but the upper_body_detector didn't work. It always said that spender/perception__internal/people_detection/rgbd_front_top/upper_body_detector: process has died for the camera/driver. I have seen that someone opened a similar issue with R200, but it was still opened. Have someone successfully run the Spencer package with intel realsense camera? Or any tips?

Thanks a lot.

[Anitawuxy]

The error is following

[Kapcirt]

Hello, One quick question, what is your camera resolution ?

[Anitawuxy]

The depth image is 480320. Then I tried realsense D415,which has a depth image resolution 640480. But it also didn't work. I faced an error in building groud_plane, which is the subscription of upper body detector.

[Kapcirt]

For the realsense ZR300, the resolution should be 640*480. At some point in the main of the upper detector the values of height and width are hard coded, that's why you get the 'process has died' error. The program is trying to access values in a matrix that are not present. What error did you faced when building the ground plane ?

[Anitawuxy]

Thanks for your answer. Then is there any solution to slove this problem?

About the error for building the ground plane. It says that the RGB-D camera driver maybe not running now. I tried to rostopic echo ground plane, but no messages show up. I can upload the screen-shot of the error/warning next Tuesday. :)

[Kapcirt]

The best solution would be to use a RGB-D solution with the correct resolution. If it is not possible you can try to modify the hard coded values in the program to 320 and 240, but I'm not sure if the detection and tracking will work afterwards.

[tlind]

Hi,

indeed, the upper-body detector is tied to a fixed resolution (partly due to the upper body depth template that has a fixed resolution). So one thing you can do is to scale your input image appropriately. But then you also need to update the CameraInfo accordingly, as the K matrix is used to compute the resulting poses from bounding boxes. _(Update: The latest version of the upper-body detector, as of end of August 2020, can scale the resolution, see the two provided config files for Asus and Kinect v2 with different resolution, and also note the depth_factor)._

The ground plane error results from the fact that your TF transform from base_footprint into the camera frame is missing. You should check if the frames mentioned in the error message correspond to the frame IDs that you are using in your system. Try using e.g. rosrun tf view_frames or rosrun tf tf_monitor.

[Anitawuxy]

Thanks tlind. Which depth image resolution is used in upper_body_detector? Because i see the ASUS Xtion PRO has VGA(640480 30fps) and QVGA(320240 60fps). I am confused about which resolution I should use. Should I also use the same frame rate?

[tlind]

As can be seen here, the expected resolution (also offered by the Asus Xtion Pro and the first-generation Kinect) is VGA, 640x480: https://github.com/spencer-project/spencer_people_tracking/blob/master/detection/rgbd_detectors/rwth_upper_body_detector/config/config_Asus.inp

[Anitawuxy]

@tlind Hi tilind, as you see that the rgbf_front_top_depth_optical_frame is already connected to the rgb_front_top_link which is linked to base_footprint. But there is no value for rgbf_front_top_depth_optical_frame. Is it the reason for the warning? Do you know how to solve it? Thanks in advance.

[tlind]

Does the yellow warning message re-appear after 20 sec? Also, run tf view_frames again after you let it play for a few seconds (looks like you are playing a bagfile?).

The structure of your TF tree looks ok, but it is strange that all transforms are very old/from t=0.0.

Have you set use_sim_time to true before running any other nodes?

[Anitawuxy]

@tlind No, this warning just appeared once. I haven't set use_sim_time to true.

Now I ran tf view frames again, it showed error. Then I checked it via rosrun tf tf_monitor. Only the base_footprint and rgbd_front_top_link was updated, the other variable were always zero.

I faced another problem. I remapped from="/spencer/sensors/rgbd_front_top/depth/image_rect_raw" to="/spencer/sensors/rgbd_front_top/depth/image_rect"/, since only the raw value has data. Then I faced the error: I want to remap the registered value to depth/image_rect, but in the rostopic list there is no registered value for the depth_image. What should I do? Should I change the camera-driver to created the registered value for depth_image or something else?

Thanks very much

[bbrito]

Any news on the last issue with the encoding? I got the same problem.

[dimaw24]

I'am using the upper body detector with a realsense d435. the detector is running, but i can't seem to get any detections. Has anyone got it to work using a realsense camera?

[bbrito]

I am trying the same but no success so far

[Anitawuxy]

How about changing the frame from map to another one?

[Alex-Beh]

@bbrito I saw your fork and seem you have solved the encoding issue. May I know do you test this package successfully using realsense D435?

@dimaw24 May I know do you successful to run the package with detection result?

I have encounter the following issue which seem similar to @Anitawuxy when using D435 camera.

[tlind]

The encoding issue should hopefully be resolved now in the latest version of the upper-body detector, thanks to the contributed pull request #59. Also different resolutions and scaling of the depth values (e.g. by a factor of 1000) are supported now, see the two new config files for Asus and Kinect v2 as an example.