Downward facing T265

[vidullan]

Hi,

I'm using an Intel 265 tracking camera and am having issues when using it downward facing.

I have no issues when it is mounted forward facing. But if placed facing downwards, the translations are reported correctly but the rotation values are extremely inaccurate and sometimes seem to be coupled.

Since the y-axis is defined towards the ground, is it fair to assume that the sensor is assigning both the y-axis and the z (toward the rear of the camera) in opposite directions leading to an under-defined coordinate frame?

Is this a known issue?

[SlavikLiman]

Hi @vidullan , As a human, facing downwards, will have a hard time to navigate, the T265 in your case is probably also does.

1. What is your platform?
2. Is there a specific reason you decided to install it facing the ground?
3. How far is the T265 located from the ground?

[patrickpoirier51]

@SlavikLiman
I presume this is the standard use case for a Drone Platform flying from 0 to 50 Meter outdoor.
There are many UAV SLAM systems using this configuration.

[vidullan]

Hi @SlavikLiman, I'm not sure I entirely agree with your comparison with humans and the T265. Sure, we as humans will have a hard time navigating facing downwards if facing a carpet or something non-descript, but imagine a case where there was a nice pattern printed on the floor. In that case if we knew the pattern we would find it just as easy (barring a stiff neck from looking down all the time).

The camera theoretically should not have a problem if the ground has distinguishable features for V-Slam. The problem here is that the implementation does not take into account that the camera can be pointed towards the ground and is messing up the coordinate frames when started in that orientation. I was wondering if the Intel engineers/whoever writes the firmware would be able to fix this.

@patrickpoirier51 is correct in that this is a standard use case for a drone as there usually are better features to be found pointing towards the ground rather than at the horizon when flying along.

[radfordi]

Hi @vidullan,

You are exactly correct that the initial "yaw" of the world coordinates will be "random" when you have a downward facing camera. The system will work just fine, but you will not be able to predict which direction is forward. We have a way to handle this internally by allowing the user to define the forward direction in device's body coordinate system and then to define which direction in the world coordinates it could be lined up with initially. e.g. the "pointing out of the top of the camera is the initial forward direction" and it "should point in the world x direction initially". The T265 accepts this kind of world coordinate definition internally, but we haven't exposed it yet. I'll move it up on the priority list.

We are also working on a change to make tracking more robust in the downward facing camera case. There's nothing inherently worse about the downward facing setup; in fact it's what I would recommend for a drone.

Thanks for your feedback.

[patrickpoirier51]

@radfordi That would be greatly appreciated as we are presently working as a Google Summer Of Code project on the T265. We can demonstrate indoor flying with Ros Based Forward Facing T265 running ArduPilot on a 450 size drone and we are getting ready to flip camera for a downward facing indoor and outdoor experiments.

[radfordi]

@patrickpoirier51, until we have a general setting for this, you can get a semi-consistent yaw by starting on a slightly inclined plane.

[patrickpoirier51]

@radfordi , so you mean having the camera looking down at an angle of 45 Deg. or less in order to get some consistent gravity vector on the Y axis ?

[radfordi]

@patrickpoirier51, no, I meant having camera face exactly down with respect to the drone body, but slightly tilting the drone itself when you start streaming poses. You could even point the camera where you want, start stream poses, then set it down.

The direction the camera is looking at start defines the -z world axis, so a slight tilt will give that a more consistent direction. However, even if the device is looking perfectly downward and we have to pick a random direction for -z, the poses and orientation should always be self-consistent modulo that initial random yaw.

[gitak2019]

I too would like to use the t265 with a downward orientation. I ended up mounting them on two drones with a slight tilt so that it initialized consistently at power up. It would be easier to flat mount the t265 though on future builds. So, a solution that allowed downward facing mounting with a consistent initialization would be appreciated.

[vidullan]

Hi @radfordi ,

So I tried to take the approach of initializing the camera with the drone tilted when it is mounted downward facing but it does not seem to work.

I checked the values using the provided example script: https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/t265_rpy.py

The roll, pitch, yaw values reported are good when it is kept titled, but as the drone is placed back on ground causing pitch to approach -90 degress, there is a coupling where roll and yaw both increase and the values I assume are being reported in an inertial frame that has changed. When the drone is tiled back again reducing pitch to greater than -90 degrees, the inertial frame switches back to the original camera initialized frame.

This makes it unusable as the inertial frame keeps shifting based on inclination. Not sure if I misunderstood you, but hopefully the internal fix is exposed soon.

[vidullan]

@radfordi Also are you aware if the tracking is degraded if there are moving objects in the frame? Due to the fisheye lenses, the front left and right propellers are visible in the frame.

Sometimes we noticed that the x,y,z, values zero themselves losing track of the drone during forward motion. Not sure if this is a known issue with the camera, or something else wrong on our end. This is one of the reasons why we wanted to downward mount it.

[radfordi]

Hi @vidullan,

The roll/pitch/yaw coordinates output by t265_rpy.py are designed to work with a forward facing camera. They have a singularity for a downward facing camera. You'll want to tweak the script with that in mind based on how you have mounted your camera.

Here's a change to t265_rpy.py that gives a consistent roll/pitch/yaw when I start facing forward and then point the camera downward.

Yes, moving objects can degrade performance, but the T265 is robust to a certain amount of motion.

[RealSenseCustomerSupport]

---

Hi @vidullan,

Any update? Is the intermediate solution provided by @radfordi help your issue?

[patrickpoirier51]

I can confirm that initializing system with an angle works ok for us https://files.gitter.im/ArduPilot/VisionProjects/w2Tt/image.png

[radfordi]

Thanks for confirming @patrickpoirier51.

[vidullan]

@RealSenseCustomerSupport Hi, thanks for checking in. Since we would have to initialize the system at an angle to have it downward facing we have given up on that till a fix that allows us to power it up downward facing is issued.

In the meantime we have mounted it at a 45 degree angle facing the ground and that seems to work well enough for our purposes.

[patrickpoirier51]

@radfordi Jim please note that in order to get the camera facing down with ArduPilot , @hoangthien94 did some modification the the Rotation Matrix,

# Downfacing, USB port to the right
H_aeroRef_T265Ref = np.array([[0,0,-1,0],[1,0,0,0],[0,-1,0,0],[0,0,0,1]])
H_T265body_aeroBody = np.array([[0,1,0,0],[1,0,0,0],[0,0,-1,0],[0,0,0,1]])

you can look at his script here: https://github.com/hoangthien94/vision_to_mavros/blob/master/scripts/t265_to_mavlink.py

[patrickpoirier51]

Yes it does, and its now fully documented as an ArduPilot WIKI http://ardupilot.org/copter/docs/common-vio-tracking-camera.html

Its quite easy using the python script as the camera position (Forward -vs- Down Looking) is a parameter that can be passed as input arguments from the command line.

Enjoy !!

[thien94]

@vizzbee Just to add to @patrickpoirier51's answer: ArduPilot now supports T265 in forward/downward facing configuration, both with and without ROS. The wikis are available for step-by-step instructions on how to get the system up and running:

• Wiki for T265 + ROS + ArduPilot
• Wiki for T265 + ArduPilot

For the sake of brevity, explanation of the system in the wikis are kept short. More in-depth explanation as well as discussion/asking for any advices when something is not working (ArduPilot-related), you may refer to the following blog posts and ArduPilot Vision Gitter channel :

• T265 + ROS + ArduPilot: part 1, part 2, part 3.
• T265 + ArduPilot: part 4, part 5.

Hope this helps.

[mzahana]

@patrickpoirier51 just out of curiosity, did you do outdoor experiments with T265? How was the performance? The localization i got outdoors is really worse than indoors. I am not sure if there are any specific setting for it to work good outdoor.

[neilyoung]

Same experience here. One plausible explanation in another thread was the lack of features at short range. In my Tests the drift was very large but in all cases the ranges were too short. There was no single case in which a distance was measured longer than in reality.

[patrickpoirier51]

@mzahana Hello, long time no talk :-)

I manage to make it work on a specific configuration:

• Looking Down
• A '' skirt'' around the unit acting as a Sun Shade
• Good feastures on the ground @ 15 Meters

Here is the result https://discuss.ardupilot.org/t/integration-of-ardupilot-and-vio-tracking-camera-part-4-non-ros-bridge-to-mavlink-in-python/44001/28

[mzahana]

@patrickpoirier51 YES! It’s been a while. I have been through some transitions :) and it is nice to chat again. Very nice work on the T265. I am using it with PX4 and it’s straight forward to integrate it through MAVROS. Works nicely indoors. In outdoors, I experienced considerable drift, but did not really have the time to investigate or improve as I was working on different VIO setups. I guess there are 2 lessons I see here.

• Adjusting the orientation of T265 to capture more features outdoors
• Avoiding excessive sun light by for example, using the nice trick you have of the “skirt” to provide sun shade.

Thanks for the information. Will try that after my vacation!

[neilyoung]

There is not much wrong with this cam at 20 x 20 m, and maybe also at 20 x 20 x 20. One need to go farther to see issues.

[partlygloudy]

We have a way to handle this internally by allowing the user to define the forward direction in device's body coordinate system and then to define which direction in the world coordinates it could be lined up with initially. e.g. the "pointing out of the top of the camera is the initial forward direction" and it "should point in the world x direction initially". The T265 accepts this kind of world coordinate definition internally, but we haven't exposed it yet. I'll move it up on the priority list.

@radfordi Any update on when this functionality may become available? My team has had some success initializing the t265 at an angle before orienting it to face the ground, however, for the particular application we're working on, that won't be an option. We know the camera will always be facing the ground on startup, but we won't be able to touch or move the vehicle in any way. The functionality you describe here is exactly what we're looking for.

[dookei]

Hi @radfordi just pinging. Any news? We use upward config and also have the same issue when in downward. Sorry just tried the last 0.2.0.951 firmware and everything is fine now. Thanks you!

[RealSenseSupport]

Thank you for highlighting the need for updates to accommodate downward facing configuration. We have moved our focus to our next generation of products and consequently, we will not be addressing this any further additions to the latest released FW for downward facing configuration in the T265.

[DiegoHerrera1890]

Hello, dear community. I am facing an issue with the T265 camera. May you guys help me? Thank you. Here is the link to my issue: https://github.com/IntelRealSense/librealsense/issues/9933