IntelRealSense / librealsense

Intel® RealSense™ SDK
https://www.intelrealsense.com/
Apache License 2.0
7.6k stars 4.83k forks source link

T265 ASIC role #5874

Closed talbenach closed 4 years ago

talbenach commented 4 years ago

The T265 datasheet states that it equipped with an ASIC for on board calculations. I could not find details on what these calculations are and what is the output of the camera to the computer.

MartyG-RealSense commented 4 years ago

In the context of the T265 data sheet, the ASIC in the T265 is the Movidius Myriad 2 VPU. Below is a quote and image from the official T265 promotional information page.


The Intel Movidius Myriad 2 VPU is a system-on-chip component that is purpose-built for image processing and computer vision at very high performance per watt, including for space-constrained implementations. Its key architectural features are the following:

• Vector processor cores are optimized for machine vision workloads. • Hardware accelerators increase throughput for imaging and computer vision. • General-purpose RISC CPU cores coordinate and direct workloads and interaction with external systems.

image

To quote Intel's guide to getting IMU data:


T265 redirects IMU readings into an Intel Movidius Myriad 2 Vision Processing Unit (VPU). The inertial sensor data is also complemented by video from two fisheye monochrome sensors that are fed into the VPU as well.

The VPU employed with T265 is capable of fusing inertial and video sensors to produce a reliable and persistent six degrees of freedom (6DOF) pose, offloading the CPU-intensive calculations from the host PC.


This processing of data in the T265's onboard Movidius VPU is different from the IMU implementation on the D435i, which delivers the raw IMU data directly to the host PC,

talbenach commented 4 years ago

It appears that the VPU is used to calculate the camera’s 6DoF. Does it play any role in tracking? Can we access the VPU to aid in tracking and reduce load from the host PC?

From: MartyG-RealSense notifications@github.com Sent: Tuesday, February 18, 2020 6:01 PM To: IntelRealSense/librealsense librealsense@noreply.github.com Cc: Noam Levy noaml@brainsgate.com; Author author@noreply.github.com Subject: Re: [IntelRealSense/librealsense] T265 ASIC role (#5874)

In the context of the T265 data sheet, the ASIC in the T265 is the Movidius Myriad 2 VPU. Below is a quote and image from the official T265 promotional information page.


The Intel Movidius Myriad 2 VPU is a system-on-chip component that is purpose-built for image processing and computer vision at very high performance per watt, including for space-constrained implementations. Its key architectural features are the following:

• Vector processor cores are optimized for machine vision workloads. • Hardware accelerators increase throughput for imaging and computer vision. • General-purpose RISC CPU cores coordinate and direct workloads and interaction with external systems.

[image]https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fuser-images.githubusercontent.com%2F41145062%2F74752886-3dca5200-5267-11ea-9d6c-734a179e470f.png&data=01%7C01%7Cnoaml%40brainsgate.com%7Cd141446564454a25770608d7b48bafff%7Cba1b9a8211954a63b7107dab2a24ddaf%7C0&sdata=Ju%2Bx6SGDhGyLc9JQF4gEGlgYmpRQ3%2BImV0cdij6VObA%3D&reserved=0

To quote Intel's guide to getting IMU data:


T265 redirects IMU readings into an Intel Movidius Myriad 2 Vision Processing Unit (VPU). The inertial sensor data is also complemented by video from two fisheye monochrome sensors that are fed into the VPU as well.

The VPU employed with T265 is capable of fusing inertial and video sensors to produce a reliable and persistent six degrees of freedom (6DOF) pose, offloading the CPU-intensive calculations from the host PC.


This processing of data in the T265's onboard Movidius VPU is different from the IMU implementation on the D435i, which delivers the raw IMU data directly to the host PC,

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FIntelRealSense%2Flibrealsense%2Fissues%2F5874%3Femail_source%3Dnotifications%26email_token%3DAGGZS4QY5V7YLD43N7CRJCTRDQAZ7A5CNFSM4KXGEBZ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMCRBTQ%23issuecomment-587534542&data=01%7C01%7Cnoaml%40brainsgate.com%7Cd141446564454a25770608d7b48bafff%7Cba1b9a8211954a63b7107dab2a24ddaf%7C0&sdata=5so0X56cR52fhaUVlrSY7TWoWrlRT36dIIlkykmpD9g%3D&reserved=0, or unsubscribehttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAGGZS4TGJQEGYPBHERGQDPLRDQAZ7ANCNFSM4KXGEBZQ&data=01%7C01%7Cnoaml%40brainsgate.com%7Cd141446564454a25770608d7b48bafff%7Cba1b9a8211954a63b7107dab2a24ddaf%7C0&sdata=jGu7l687qrIU816PNPG7y6ioVUsB5AtfTG0FNcQHYyg%3D&reserved=0.

MartyG-RealSense commented 4 years ago

As far as I know, all the data on the T265 is processed through the Movidius VPU by default. The link below has an official Intel case-study for using the T265 that describes the VPU's role in tracking in the second to last section near the bottom of the page. The section is headed "Host connectivity and integration into devices".

https://www.intelrealsense.com/visual-inertial-tracking-case-study/

RealSenseCustomerSupport commented 4 years ago

To add to MartyG-RealSense, T265 is a closed system SLAM solution. The Movidius VPU is used to take input from the IMU that is onboard and the fisheye cameras and utilize an Intel proprietary SLAM algorithm to calculate it's position in space. The final output to host system is pose data. One can also get raw fisheye images and raw IMU (accel and gyro) data to the host system for host side processing if developer so needs that information. We do not allow for developers to use the Movidius VPU to run their own FW/Algorithm. T265 runs it's own FW/Algorithm.

Let us know if you need anything else on this topic.

talbenach commented 4 years ago

Thank you very much for this clarification. I assume the answer to the following is no, but I want to make sure I am not missing anything. Is there access to feature recognition data generated by the Movidius? E.g. corners recognition?

Thank you,

From: RealSense Customer Support notifications@github.com Sent: Saturday, February 22, 2020 1:59 AM To: IntelRealSense/librealsense librealsense@noreply.github.com Cc: Noam Levy noaml@brainsgate.com; Author author@noreply.github.com Subject: Re: [IntelRealSense/librealsense] T265 ASIC role (#5874)


To add to MartyG-RealSense, T265 is a closed system SLAM solution. The Movidius VPU is used to take input from the IMU that is onboard and the fisheye cameras and utilize an Intel proprietary SLAM algorithm to calculate it's position in space. The final output to host system is pose data. One can also get raw fisheye images and raw IMU (accel and gyro) data to the host system for host side processing if developer so needs that information. We do not allow for developers to use the Movidius VPU to run their own FW/Algorithm. T265 runs it's own FW/Algorithm.

Let us know if you need anything else on this topic.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FIntelRealSense%2Flibrealsense%2Fissues%2F5874%3Femail_source%3Dnotifications%26email_token%3DAGGZS4QCITDGHLIT2BGOV6DREBTFJA5CNFSM4KXGEBZ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMUPHFI%23issuecomment-589886357&data=01%7C01%7Cnoaml%40brainsgate.com%7Cbdeebb90abd5416b400508d7b72a1053%7Cba1b9a8211954a63b7107dab2a24ddaf%7C0&sdata=a4TSwP4Sn9tbMFaQtr4v65iPYmVChufKq9mWDWMbHcs%3D&reserved=0, or unsubscribehttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAGGZS4Q65VS7GVZ4GOFJWCLREBTFJANCNFSM4KXGEBZQ&data=01%7C01%7Cnoaml%40brainsgate.com%7Cbdeebb90abd5416b400508d7b72a1053%7Cba1b9a8211954a63b7107dab2a24ddaf%7C0&sdata=Y%2BvgeT%2B308iAEbfN8TJWWYV8myRzcLP6nAP75CZcH%2Fs%3D&reserved=0.

MartyG-RealSense commented 4 years ago

Intel's guide to Visual SLAM and the T265 has the following quote:


This device is intended for customers looking for a hassle-free stand-alone SLAM system. For those who want to develop or deploy their own tracking algorithms, they can obtain the raw data (cameras and IMU) from the T265, or build on top of our Intel RealSense Depth Camera D400 family which provide similar sensor capabilities (albeit with smaller FOV) and also output depth maps.

However, note that any SLAM algorithm would need to be run on the host platform and not embedded and this will increase the weight, power budget and cost of the target platform.


The Visual SLAM guide is available at the link below.

https://dev.intelrealsense.com/docs/intel-realsensetm-visual-slam-and-the-t265-tracking-camera

It discusses feature detection algorithms such as corner detection in a general way not specific to the T265, quoting algorithms such as Harris Corner and Shi-Tomasi Corner. The image below from the guide shows corner detection using the Harris Corner algorithm,

image

RealSenseCustomerSupport commented 4 years ago

No access to feature data from the Movidius integrated in T265.

RealSenseCustomerSupport commented 4 years ago

Is there anything else needed on this thread?

If we don’t hear from you in 7 days, this issue will be closed.