Running DREAM ros node in multi-frame mode

[mohakbhardwaj]

Hi, I am trying to run DREAM in multi-frame mode with ROS using launch_dream_ros.py on a Franka Panda robot. However, in the code it appears that the PnP solver is only called when running in single-frame mode as shown below.

     # Solve PNP (if in single frame mode)
        if single_frame_mode and found_kp_projs_net_input is not None:
            (
                kp_projs_raw_good_sample,
                kp_positions_good_sample,
            ) = dream_ros.keypoint_correspondences(found_kp_projs_net_input)
            if (
                kp_projs_raw_good_sample is not None
                and kp_positions_good_sample is not None
            ):
                dream_ros.solve_pnp_buffer(
                    kp_projs_raw_good_sample, kp_positions_good_sample
                )

Could you let me know how I can setup PnP with multi-frame mode? I would like to move the robot to different poses and obtain an accurate estimate of camera to robot base transform.

[tabula-rosa]

Hey Mohak, thanks for your interest in our work!

For multi-frame mode, I believe most of the functionality should be there, but user interaction is required to decide which image frames to keep, and when to clear the keypoint buffers.

I think these instructions should work. Could you please confirm whether this works for you, or if you ran into any issues?

• Change this line to single_frame_mode = False
• Launch the node as usual
• Each time you want to capture a frame, call the ROS service through the following terminal command: rosservice call /dream/capture_frame
• When you want to start over, clear the keypoint buffers by calling the ROS service: rosservice call /dream/clear_buffer

An important assumption in multi-frame mode is that the camera-robot transform is expected to be static. (This isn't the case with single-frame mode, as it runs inference at each frame.) So, the use case you've described (moving the robot to different poses) should provide a more accurate calibration estimate than in single-frame mode alone. Hope that helps!

[tabula-rosa]

I'll go ahead and close this issue for now, but in case you still have difficulties, please comment and I'd be happy to re-open the issue to help troubleshoot!