search

IntelRealSense / realsense-ros

ROS Wrapper for Intel(R) RealSense(TM) Cameras
http://wiki.ros.org/RealSense
Apache License 2.0
2.51k stars 1.74k forks source link

Working with Intel realsense T265 tracking camera and D435i Depth camera together? #1011

Closed isirana closed 3 years ago

isirana commented 4 years ago

Hi, Did anyone try to implement SLAM with combining T265 camera and D435i depth camera? When I try to use both these cameras with ROS, most of the times It gives this kind of an error

11:27:58.880 [13072] [E] Device-5260: Error: Failed to claim USB interface. LIBUSB_ERROR_CODE: 0xFFFFFFFA (LIBUSB_ERROR_BUSY)

Sometimes I managed to get the data from both the cameras to the rviz without any error. but the problem is , I couldn't manage to get any steady connection from both these camera when I use with ROS.

(please note: I can access both these cameras from realsense viewer without any issue.)

Does anyone know the reason for that?

MartyG-RealSense commented 4 years ago

Intel produced a YouTube presentation on using T265 and D435 together, with the T265 providing navigation and the D435 providing obstacle avoidance. The part about D435 starts at 17 minutes 15 seconds into the video.

https://www.youtube.com/watch?v=62vm0_RZ1nU

A shorter YouTube clip on the subject of combining T265 and D435 is here:

https://www.youtube.com/watch?v=LBMsWJJxLXQ

One user who combined T265 and D435 for mapping found that they could achieve what they needed to in ROS with minimal editing of the realsense-ros launch file.

https://github.com/IntelRealSense/librealsense/issues/4983#issuecomment-538675614

To visualize the T265 pose output and frames in Rviz, the following launch instruction can be used:

roslaunch realsense2_camera demo_t265.launch

doronhi commented 4 years ago

On my machines, when I run roslaunch realsense2_camera rs_d400_and_t265.launch the message "Failed to claim USB interface" is printed many time by the t265 node during the initial second or so, until the d435i node releases it and then the t265 device is claimed and everything works fine. I can then run RViz, set the "Fixed Frame" to "t265_pose_frame" and see the pointcloud projected in the right direction, according to the t265's orientation reports. Could you elaborate what kind of hardware you are using? What OS? versions? etc.

abdellaui commented 4 years ago

it works for me! look at: https://github.com/abdellaui/ros_intel_realsense_t265_d435

isirana commented 4 years ago

Hi, I managed to get the data from both cameras to rviz by running roslaunch realsense2_camera rs_camera.launch camera:=cam_1 serial_no:=845412110596 roslaunch realsense2_camera rs_camera.launch camera:=cam_2 serial_no:=909512071732

separately, but still D435i camera gives a warning like this:

WARNING [547071455616] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61

Also, when I run the _roslaunch realsense2_camera rs_d400_andt265.launch , D435i camera gives the "Failed to claim USB interface" and it doesn't give any data from the rviz.

I am using Nvidia Tx1 with jetpack 4.2 and ubuntu 18.04 LTS version.

MartyG-RealSense commented 4 years ago

@doronhi is likely the best person to assist with this question. Having said that, the ability to launch by USB port ID instead of serial number was recently added, so you could explore that possibility

https://support.intelrealsense.com/hc/en-us/community/posts/360037088214-Information-on-choosing-a-device-in-the-ROS-wrapper-by-USB-port-ID

RealSenseCustomerSupport commented 4 years ago

Hi @isirana

Were you able to get past any issues you were having here? Is there any further assistance necessary?

isirana commented 4 years ago

Hi @isirana

Were you able to get past any issues you were having here? Is there any further assistance necessary?

yes, I am still having problem when connecting both the cameras together to perform autonomus navigation task with ROS. Some times I was able to connect both the cameras with ROS by connecting the D435i camera after T265 camera node is up.However, USB overflow warning is given after I run it some times. I am using a Nvidia Jetson TX1 platform and the cameras are connected through a USB hub. When I use the realsense viewer I can use both the cameras without any issue.But the problem comes when I connect the cameras with ROS.

sadransh commented 4 years ago

I remember I had no issue with version 2.29 and 2.30 now it is updated and I am expriencing similar problems

sadransh commented 4 years ago

@isirana Do you still have the issue? any idea?

sadransh commented 4 years ago

have you tried rebuilding the librealsense ?( instead of installing from the repository?) ( just a guess)

sadransh commented 4 years ago

@doronhi do you have any idea on why this happens? Is there any temporary fix for this? I have installed library using repository. Is there any possibility of fixing this problem if I re build the lib on device? Any other guess on this? I have time to test.

GregCannon63 commented 4 years ago

I have the same issue. I've tested on multiple machines to rule out USB hardware specific issues. Running Ubuntu 18.04 5.3.0-40-generic #32~18.04.1-Ubuntu SMP x86_64 GNU/Linux

Ultimately, I want to use ROS2. I tried ROS2 installed independently I reinstalled using the RobotDevKit.

ros2 launch realsense_examples rs_t265_and_d400.launch.py produces

[INFO] [launch]: All log files can be found below /home/csgp/.ros/log/2020-03-10-16-02-18-896227-Happy-24679
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [rviz2-1]: process started with pid [24696]
[INFO] [static_transform_publisher-2]: process started with pid [24697]
[INFO] [realsense_node-3]: process started with pid [24698]
[INFO] [realsense_node-4]: process started with pid [24699]
[realsense_node-3] terminate called after throwing an instance of 'rs2::error'
[realsense_node-3]   what():  Couldn't resolve requests
[ERROR] [realsense_node-3]: process has died [pid 24698, exit code -6, cmd '/opt/robot_devkit/perception/realsense_node/lib/realsense_node/realsense_node __ns:=/t265 __params:=/tmp/launch_params_gs0mbi5l /t265/camera/odom/sample:=/odom'].

~/robot_devkit/rdk_ws$ ros2 run realsense_node realsense_node params:=ros2 pkg prefix realsense_examples/share/realsense_examples/config/d435.yaml ns:=/d435 works fine

[INFO] [d435.camera]: Device with serial number 843112070975 was found.
[INFO] [d435.camera]: Create a node for D435i Camera

~/robot_devkit/rdk_ws$ ros2 run realsense_node realsense_node params:=ros2 pkg prefix realsense_examples/share/realsense_examples/config/t265.yaml ns:=/t265 produces

[INFO] [t265.camera]: Device with serial number 843112070975 was found.
[INFO] [t265.camera]: Device with serial number 943222110250 was found.
[INFO] [t265.camera]: Create a node for T265 Camera
terminate called after throwing an instance of 'rs2::error'
  what():  Couldn't resolve requests
abdellaui commented 4 years ago

im wondering that ros is running on ubuntu 16.04 upward. the first sentence of the installtion-guide is, that ros is only supported on Wily (Ubuntu 15.10), Xenial (Ubuntu 16.04) and Jessie (Debian 8) for debian packages. maybe you install ubuntu 16.04 first and run it like here, it should work!

GregCannon63 commented 4 years ago

As a test, I modified rs_tracking_and_depth.cpp as follows: I replaced for (auto&& dev : ctx.query_devices()) { rs2::pipeline pipe(ctx); rs2::config cfg; cfg.enable_device(dev.get_info(RS2_CAMERA_INFO_SERIAL_NUMBER)); pipe.start(cfg); pipelines.emplace_back(pipe); } with

    rs2::pipeline pipeD435(ctx);
    rs2::config cfgD435;
    cfgD435.enable_device("843112070975");
    pipeD435.start(cfgD435);
    pipelines.emplace_back(pipeD435);
    rs2::pipeline pipeT265(ctx);
    rs2::config cfgT265;
    //cfgT265.enable_stream(RS2_STREAM_POSE, RS2_FORMAT_6DOF);
    cfgT265.enable_device("943222110250");
    pipeT265.start(cfgT265);
    pipelines.emplace_back(pipeT265);`

If I try to enable the T265 using cfgT265.enable_stream(RS2_STREAM_POSE, RS2_FORMAT_6DOF) it works just fine. If I try to enable using the serial number cfgT265.enable_device("943222110250");` it fails.

That's because in device_hub.cpp, std::shared_ptr<device_interface> device_hub::create_device(const std::string& serial, bool cycle_devices) always retrieves the D435i serial number for comparison. ` device_hub.cpp (81) serial:943222110250 new_serial:843112070975

device_hub.cpp (81) serial:943222110250 new_serial:843112070975

device_hub.cpp (81) serial:943222110250 new_serial:843112070975

device_hub.cpp (81) serial:943222110250 new_serial:843112070975

device_hub.cpp (81) serial:943222110250 new_serial:843112070975

device_hub.cpp (81) serial:943222110250 new_serial:843112070975

device_hub.cpp (81) serial:943222110250 new_serial:843112070975

device_hub.cpp (81) serial:943222110250 new_serial:843112070975

device_hub.cpp (81) serial:943222110250 new_serial:843112070975

RealSense error calling rs2_pipeline_start_with_config(pipe:0x555555d132b0, config:0x555555d0bf80): No device connected ` The T265 does not appear to be in the device list. It shows in rs-enaumerate-devices, but however device_hub.cpp is handling it, it doesn't see it. I don't know how to fix that, but that's where my troubleshooting has led so far.

GregCannon63 commented 4 years ago

@abdellaui, ROS 1 will run on Ubuntu Bionic (18.04) http://wiki.ros.org/melodic/Installation/Ubuntu. However, I'm running ROS2.

That being said, this is a librealsense issue, not a ROS or ROS2 issue. The code I've posted above for testing uses the Realsense command line examples to test.

gramss commented 4 years ago

@GregCannon63 did you have success running the T265 with ROS2 via cfgT265.enable_stream(RS2_STREAM_POSE, RS2_FORMAT_6DOF)? Is there any documentation running it with ROS2 from you or somebody else? Thank you in advise! :)

GregCannon63 commented 4 years ago

@mrlowalowa, yes, enabling by stream works with the T265. But enabling by serial number fails. The ROS2 docs are here I found a hint from another project here the led to me to find that device_hub.cpp has a method called filter_by_vid(), where any device that isn't a uvc (USB video camera) and has the assumed Intel vid (VendorID) of '8086' is discarded. Well, the T265 isn't a uvc and has a vid of "8087" and it has a pid (Product ID) of "0x0B37".

So during a search by serial number, the T265 devices get filtered out. I modified the method as follows: ` std::vector<std::shared_ptr> filter_by_vid(std::vector<std::shared_ptr> devices , int vid)

{

    std::vector<std::shared_ptr<device_info>> result;

    for (auto dev : devices)

    {

        bool filtered = false;

        auto data = dev->get_device_data();

        for (const auto& uvc : data.uvc_devices)

        {

            if (uvc.vid == vid || vid == 0)

            {

                result.push_back(dev);

                filtered = true;

                break;

            }

        }

//Begin: Added to allow T256 to be found by serial number

        if (!filtered)

        {

            uint16_t pid = 0x0B37;

            auto tm2_pid = filter_by_product(data.usb_devices, { pid });

            if (tm2_pid.size())

            {

                for (const auto& usb : data.usb_devices)

                {

                    if (usb.pid == pid)

                    {

                        result.push_back(dev);

                        filtered = true;

                        break;

                    }

                }

            }

        }

//End: Added to allow T256 to be found by serial number 

    }

    return result;

}

` This works when searching by serial in code, but this doesn't fix the issue when a .yaml config file is used. I'm looking into that today.

The bad news is that this has taken so long to debug, and has had so little support from Intel, that my managers are probably going to force me to drop the project and move on.

RealSenseSupport commented 3 years ago

Thank you for highlighting the issue outlined regarding T265. We have moved our focus to our next generation of products and consequently, we will not be addressing this issue in the T265.