T265 is loosing position under vibrations

[urock]

---

Required Info
Camera Model	T265
Firmware Version	0.0.18.5715
Operating System & Version	Ubuntu 16
Platform	PC
SDK Version	2.22.0
Segment	Robot

Issue Description

Hi everybody!

I have noticed when using T265 under vibrations above certain threshold it tends to loose its position very fast. Even if you shake camera in hand, not to mention using it on drone as I am doing. This video also shows this issue https://www.youtube.com/watch?v=GhHvuAoFC6I

I hope firmware developers could do smth with it, right now it is impossible to use T265 under vibrations.

Meanwhile I will try to use vibration vibration damping pads.

[MartyG-RealSense]

Vibration may be more of a problem if you are using SLAM than it is with depth sensing on the 400 Series cameras (as the 400 Series has very robust stereo vision algorithms). On the T265, 'wheel odometry' is used as a stabilizing mechanism.

https://github.com/IntelRealSense/librealsense/blob/master/doc/t265.md#what-is-wheel-odometry-and-how-does-it-help-t265-navigate

Wheel odometry is not likely to be useful on a flying drone though if there is no turning of a robot's wheels involved.

There is an example on YouTube of T265 being used with a flying drone and ArduPilot that was published last week.

https://www.youtube.com/watch?v=YZm6bmGDB5U

[urock]

I've just wanted to be sure that developers are aware of this issue. Is there anything that could be done on firmware side of T265 to fix it? Is it on dev team roadmap? Thanks

[patrickpoirier51]

Yes VIO is quite sensible to vibration , Here a demonstration with a fixed mounted camera-imu system on ROVIO: https://www.youtube.com/watch?v=QrTb8YpC-Lg

Since then , I mount my systems on Damping Reduction Platforms ( 2$ @ Banggood), here is the T265 mounted on such a Damper Platform:

[patrickpoirier51]

@MartyG-RealSense video is from Thien Nguyen a PHD Student , I am mentoring on GSOC ( Goodle Summer Of Code) project this summer. You can watch his progress here: https://discuss.ardupilot.org/t/gsoc-2019-integration-of-ardupilot-and-vio-tracking-camera-for-gps-less-localization-and-navigation/42394

I asked him to upload on the RealSense forum once completed (2-3 weeks), we will certainly get in touch with you as you are the champion of the forum ;-)

[urock]

@patrickpoirier51 can you pls share what have changed after you used damping reducing platform? Is t265 performing better now?

[patrickpoirier51]

I haven't done comparison yet on T265, but on the setup showned above , that was the only way to make it work (as you can see the EKF was drifting until loosing track)

Vibration tests is part of the experimentation that Thien Nguyen will be doing this week. He has a fixed mounted T265 on his setup and he will be inducing various degree of vibrations using an unbalanced propeller. We will upload Lab report here once completed.

[radfordi]

@urock, @patrickpoirier51, all our internal testing has been on Falcon 8s, and I haven't seen any issues with vibration. I can imagine that smaller drones might have different vibration characteristics. You should be able to filter the motor frequencies with a custom vibration insulator.

[patrickpoirier51]

Here is the Lab Report https://discuss.ardupilot.org/t/integration-of-ardupilot-and-vio-tracking-camera-part-3-indoor-autonomous-flights-and-performance-tests/43626

The T265 is fixed mounted to a 3D printed bracket, and take note that on the vibration test video, hoangthien94 had to manually hold and shake the Quadcopter in order to get the system diverge (degrade) for a short period.

[alexbarnett12]

Hi @patrickpoirier51 , I am trying to use the T265 on a UAV for autonomous flight in outdoor environments. I have performed successful indoor autonomous flight, but as soon as I go outside the T265 pose estimate diverges and the drone loses control. I am guessing it is a mixture of my vibration and the increased scale of outdoor environments, since it works extremely well inside with the same vibration levels. I will be trying different vibration damping techniques soon to see if I can get stable flight. Do you have an idea of the maximum vibration that the T265 can operate in?

In other posts you have said you've obtained accurate scale up to 60 ft with the T265. I am curious about your mounting configuration - was it forward or downward facing? Due to size constraints, I have to mount the T265 on the side of the drone at a 45 degree angle. Do you think that mounting configuration could be causing some of my issues? I am confident my pose transform is correct as I have successfully flown indoors. I have attached a picture of the drone and vibration on the T265 in m/s^2 in the UAV body coordinate system (x = forward, y = right, z = down).

[patrickpoirier51]

@alexbarnett12 Going outdoor is definitely another ball game. Because of the fisheye lens the T265 is prone to light saturation when it gets sunlight. Look here for my configuration, it has the sensor looking down and I added a "skirt" to reduce exposure https://github.com/IntelRealSense/librealsense/issues/4681#issuecomment-528360702 You need good features on ground and stable platform https://discuss.ardupilot.org/t/integration-of-ardupilot-and-vio-tracking-camera-part-4-non-ros-bridge-to-mavlink-in-python/44001/28

I recommend you get a short angled USB3 cable because most of vibrations should comme from cable loop that catches airflow

[radfordi]

@alexbarnett12, I would concentrate on mechanical dampening. The T265 gets scale from two places, one is stereo and the other is the accelerometer. When you are 60ft up in a drone, stereo is close to useless, as the baseline is too small compared to that distance, so we have to rely on the accelerometer. I see acceleration noise greater than 1g! That's going to cause problems for scale recovery.

[urock]

Hi guys, let me put mine dime into the discussion. We have performed tests of T265 on vibration stand going though frequencies and acceleration amplitude in [2 - 150 Hz] and [0.5 - 8 g] measuring point when T265 would start loosing position. Results are the following

You can see that with increasing frequency acceleration needed for T265 to loose position decreases from 8g to 1g saturating at 50 Hz. Saturation occurs because T265 can't report acceleration faster than 50-60 Hz. But this graf definitely shows that higher vibration frequencies have stronger impact on T265.

Also I can report that we are flying with T265 without any dampening pads, it is firmly fixed on drone frame. But we have put a lot of efforts to flight vibrations on the frame itself. We are using best carbon props, motor mounts are firmly fixed, frame has to be solid without any loose parts.

[MathiasThor]

@alexbarnett12 would you be able to share your pose transformation for the 45 degree angled mount?

[alexbarnett12]

Update from my last post: I was able to achieve stable outdoor flight by switching the Realsense mounting configuration to downward or forward facing. I did not have to make any changes to the level of vibration reaching the Realsense. I find it quite interesting that the 45-degree mounting configuration worked well indoors but not outdoors - I don't have the expertise to explain this. It might be something that the Intel Realsense team should look into.

However, vibration is sometimes still a significant problem. Every once in a while, the Realsense will experience tracking failure upon takeoff outside. It seems completely random as I will perform a great test flight, restart the UAV, and then it will crash. I think this is probably because of my vibration levels, but with my size drone and my limited time and manufacturing capabilities, I cannot feasibly reduce the frequency or magnitude of the vibration.

Here is a video of an autonomous flight test I performed where the UAV (slowly) navigates towards a yellow tarp. I never got around to testing at very high altitudes before COVID-19, so I don't know if the Realsense would experience tracking failure at 60+ feet.

@MathiasThor I have included my 45-degree pose transformation if you would still like it. I can't recommend mounting it this way if that's what you were planning! :)

## Realsense to NED pose transformations ##
# 0: NED Origin
# 1: Realsense (RS) Origin
# 2: NED Frame
# 3: RS Frame
#
# H0_2: NED Frame rel. to NED origin (this is the final pose sent to the Pixhawk)
# H0_1: RS Origin rel. to NED Origin
# H1_3: RS Frame rel. to RS Origin (pose received from Realsense)
# H3_2: NED Frame rel. to RS Frame
#
# H0_2 = H0_1.dot(H1_3).dot(H3_2)
#
# With SLAM flight enabled and GPS disabled, NED and body frame are equivalent, since the
# UAV has no idea of true north. NED origin is defined based on the starting orientation of the UAV.

# Upside down, rotated 40 degrees, sideways mount
# H3_2 = H3_A.dot(HA_B).dot(HB_2)
# This breaks down into a 180 deg. flip about z (A), 45 deg rotation about x (B) and 
# left hand to right hand coord. system change.

H0_1 = np.array([[1, 0, 0, 0],
                 [0, 0, 1, 0],
                 [0, -1, 0, 0],
                 [0, 0, 0, 1]])

HA_3 = np.array([[-1, 0, 0, 0],
                 [0, -1, 0, 0],
                 [0, 0, 1, 0],
                 [0, 0, 0, 1]])

H3_A = tf.inverse_matrix(HA_3)

HA_B = np.array([[1, 0, 0, 0],
                 [0, np.cos(40*np.pi / 180), -np.sin(40*np.pi / 180), 0],
                 [0, np.sin(40*np.pi / 180), np.cos(40*np.pi / 180), 0],
                 [0, 0, 0, 1]])

HB_2 = np.array([[1, 0, 0, 0],
                 [0, 0, -1, 0],
                 [0, 1, 0, 0],
                 [0, 0, 0, 1]])

H3_2 = H3_A.dot(HA_B).dot(HB_2)

[RealSenseSupport]

Thank you for highlighting the issues with pose under vibrations on T265. We have moved our focus to our next generation of products and consequently, we will not be addressing this issue in the T265.

[DiegoHerrera1890]

Hello Everyone. Thank you for sharing all this information. I am facing an issue with the T265 I hope you can give some advice in order to fly outdoors. I have flown my drone indoors without problems using the T265 realsense camera. I have done many tasks for my research. I want to fly outdoors but the drone seems to lose its position and orientation. it starts to take off but the height (z position) shows negative values, in my program when it reads negative height the drone lands and disarms it, so it cannot fly. I have tried to see the possible errors in the programming but everything is fine since in the laboratory it works perfectly with the same configurations. I emphasize that the t265 camera is looking forward. I’m attaching the fisheye view of one lens. Any suggestion is welcome.

Fisheye view outdoors (show Z negatives values)

Fisheye view indoors (Fly perfectly)

Thank you so much