RS2_STREAM_POSE not supported with L515 (OR calculate pose data within the librealsense API)

[andrewhodel]

• Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):

  • Consider checking out SDK examples.
  • Have you looked in our documentations?
  • Is you question a frequently asked one?
  • Try searching our GitHub Issues (open and closed) for a similar issue.

• All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

---

Required Info
Camera Model	{ R200 / F200 / SR300 / ZR300 / D400 }
Firmware Version	(Open RealSense Viewer --> Click info)
Operating System & Version	{Win (8.1/10) / Linux (Ubuntu 14/16/17) / MacOS
Kernel Version (Linux Only)	(e.g. 4.14.13)
Platform	PC/Raspberry Pi/ NVIDIA Jetson / etc..
SDK Version	{ legacy / 2.<?>.<?> }
Language	{C/C#/labview/nodejs/opencv/pcl/python/unity }
Segment	{Robot/Smartphone/VR/AR/others }

Issue Description

<Describe your issue / question / feature request / etc..>

Why is RS2_STREAM_POSE not supported with the L515, it has an IMU.

[andrewhodel]

rs-pose does not work with the L515

[andrewhodel]

Makes no sense, I can get accel and gyro values for x,y,z but I cannot get pose data?

The pose data is derived from accel and gyro values!

[andrewhodel]

The camera that is said to be supported in the documentation, the T265 uses a 6 axis IMU (no magnetometer).

https://www.digikey.com/product-detail/en/bosch-sensortec/BMI055/828-1043-1-ND/4196679?utm_adgroup=Sensors%20%26%20Transducers&utm_source=google&utm_medium=cpc&utm_campaign=Dynamic%20Search&utm_term=&utm_content=Sensors%20%26%20Transducers&gclid=EAIaIQobChMI49aBvcDc6gIV4gl9Ch1LsAO_EAAYASAAEgLkvPD_BwE

That means that yaw is calculated in pose from the depth data.

Which means the L515 should also support pose.

[RealSenseSupport]

Hi @andrewhodel,

Our tracking camera has custom hw/sw specifically designed to return pose data. You can learn more about our T265 camera here:

https://www.intelrealsense.com/tracking-camera-t265/

Thanks

[Richard-Elliott]

Can the L515 and T265 be paired to work together in a similar fashion to how the T265 can work with the D400 series depth sensors?

[andrewhodel]

The l515 outputs all the required data that is needed to calculate pose data.

Andrew

On Tue, Jul 21, 2020 at 9:41 AM Richard-Elliott notifications@github.com wrote:

Can the L515 and T265 be paired to work together in a similar fashion to how the T265 can work with the D400 series depth sensors?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/6869#issuecomment-661938179, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFVCSNV5VFHBEVW3TWJCZLR4WZMNANCNFSM4PCSNX4A .

[Richard-Elliott]

Hi Andrew, I realise the L515 has the IMU, but it does not have the onboard calculation of pose from the fusion of the IMU data and the wide angle cameras as on the tracking camera. If using an IMU to calculate pose the result tends to drift over time if there is not some other sensor value to reset the drift. I'm sure this could be done by fusing the lidar data with the IMU data but the processing would be external to the device rather than onboard. Hence my question can the 2 devices be paired T265 with L515?

[andrewhodel]

All the devices output data in an open format.

You can combine whatever you want, 50,000 of them if you like.

The l515 has lidar for god sakes, it isn’t that difficult to calculate pose data when you have lidar and a 6dof imu on the same device.

Of course if you have 50,000 lifar cameras then I want a jet.

Andrew

On Tue, Jul 21, 2020 at 10:32 AM Richard-Elliott notifications@github.com wrote:

Hi Andrew, I realise the L515 has the IMU, but it does not have the onboard calculation of pose from the fusion of the IMU data and the wide angle cameras as on the tracking camera. If using an IMU to calculate pose the result tends to drift over time if there is not some other sensor value to reset the drift. I'm sure this could be done by fusing the lidar data with the IMU data but the processing would be external to the device rather than onboard. Hence my question can the 2 devices be paired T265 with L515?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/6869#issuecomment-661967049, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFVCSLRQMMC6NLSACUXIVTR4W7IZANCNFSM4PCSNX4A .

[Richard-Elliott]

I wanted to check the compatibility of the cameras specifically if the IR filters on the tracking camera would filter out the IR laser used on the LIDAR?

[andrewhodel]

Dude, have you not seen the transistors lately, they are crazy high amperage and fast.

There is no reason every laser pulse wouldn’t be encrypted with xor and unique for decryption.

Andrew

On Tue, Jul 21, 2020 at 10:41 AM Richard-Elliott notifications@github.com wrote:

I wanted to check the compatibility of the cameras specifically if the IR filters on the tracking camera would filter out the IR laser used on the LIDAR?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/6869#issuecomment-661972312, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFVCSIHQXUPTWDGJS5BGKTR4XALBANCNFSM4PCSNX4A .

[andrewhodel]

Do you think there is any other way they would be safe for use in stopping cars via object detection?

Thank You, Andrew Hodel

On Jul 21, 2020, at 10:41 AM, Richard-Elliott notifications@github.com wrote:

 I wanted to check the compatibility of the cameras specifically if the IR filters on the tracking camera would filter out the IR laser used on the LIDAR?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[andrewhodel]

And it is strange how off subject you get on this Richard, we are trying to work here not sit in the office.

On Tue, Jul 21, 2020 at 10:44 AM Andrew Hodel andrewhodel@gmail.com wrote:

Do you think there is any other way they would be safe for use in stopping cars via object detection?

Thank You, Andrew Hodel

On Jul 21, 2020, at 10:41 AM, Richard-Elliott notifications@github.com wrote:



I wanted to check the compatibility of the cameras specifically if the IR filters on the tracking camera would filter out the IR laser used on the LIDAR?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/6869#issuecomment-661972312, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFVCSIHQXUPTWDGJS5BGKTR4XALBANCNFSM4PCSNX4A .

[blakely3dnm]

If you derived pose from just IMU accel and gyro you would not have very good results. Including the lidar information allows for pose estimation using ICP but this is fairly expensive compared to visual odometry methods, like the t265 uses. As said above it is possible to do but it is done external to the device.

[andrewhodel]

Lidar would provide much better results for pose data in coordination with the IMU data than visual camera data would.

The code running on the l515 has access to both the IMU and lidar data.

There is no reason that the l515 shouldn’t provide the best pose data of all the realsense cameras.

Andrew

On Tue, Jul 21, 2020 at 3:55 PM blakely3dnm notifications@github.com wrote:

If you derived pose from just IMU accel and gyro you would not have very good results. Including the lidar information allows for pose estimation using ICP but this is fairly expensive compared to visual odometry methods, like the t265 uses. As said above it is possible to do but it is done external to the device.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/6869#issuecomment-662126714, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFVCSMY3TCY7ANWHSIKMQLR4YFGDANCNFSM4PCSNX4A .

[blakely3dnm]

Put a core i7 in the L515 and, sure. It should provide pose.

[andrewhodel]

Now we are talking, but there is no reason that librealsense (the api code that runs on the host) shouldn’t have an api for fetching the pose data.

You could make the argument that it shouldn’t be in the camera stream config.

On the other hand, you really only need maybe 10 pixels of data spread across the view to get the pose data streaming directly from the camera with the IMU data. That would be possible with the cpu that is already in there as pathetic as it is.

Andrew

On Tue, Jul 21, 2020 at 4:06 PM blakely3dnm notifications@github.com wrote:

Put a core i7 in the L515 and, sure. It should provide pose.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/6869#issuecomment-662130961, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFVCSKBCK63C64HNVXHPJDR4YGNNANCNFSM4PCSNX4A .

[andrewhodel]

Are you really telling me Intel would make a better product for room scanning and just not write the code for getting pose data?

I just cannot believe that.

On Tue, Jul 21, 2020 at 4:09 PM Andrew Hodel andrewhodel@gmail.com wrote:

Now we are talking, but there is no reason that librealsense (the api code that runs on the host) shouldn’t have an api for fetching the pose data.

You could make the argument that it shouldn’t be in the camera stream config.

On the other hand, you really only need maybe 10 pixels of data spread across the view to get the pose data streaming directly from the camera with the IMU data. That would be possible with the cpu that is already in there as pathetic as it is.

Andrew

On Tue, Jul 21, 2020 at 4:06 PM blakely3dnm notifications@github.com wrote:

Put a core i7 in the L515 and, sure. It should provide pose.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/6869#issuecomment-662130961, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFVCSKBCK63C64HNVXHPJDR4YGNNANCNFSM4PCSNX4A .

[blakely3dnm]

I am not familiar with any ICP odometry methods that downsample the clouds to 10 points. Maybe you should write an algorithm that does this, and I will be able to do SLAM with my raspberry pi.

[andrewhodel]

You are too familiar, you know how to hold on to 10 ropes that are tied to evenly spread points in a room and count your steps and direction to know where you are.

What is the reason Intel with all it’s abundant funding cannot accomplish this?

Andrew

On Tue, Jul 21, 2020 at 7:03 PM blakely3dnm notifications@github.com wrote:

I am not familiar with any ICP odometry methods that downsample the clouds to 10 points. Maybe you should write an algorithm that does this, and I will be able to do SLAM with my raspberry pi.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/6869#issuecomment-662182501, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFVCSLB5SRMCYSPGH6IIW3R4Y3FRANCNFSM4PCSNX4A .

[andrewhodel]

You can gain your location from the ropes alone or the steps and direction alone.

The precision would be very good when combined.

Andrew

On Tue, Jul 21, 2020 at 7:06 PM Andrew Hodel andrewhodel@gmail.com wrote:

You are too familiar, you know how to hold on to 10 ropes that are tied to evenly spread points in a room and count your steps and direction to know where you are.

What is the reason Intel with all it’s abundant funding cannot accomplish this?

Andrew

On Tue, Jul 21, 2020 at 7:03 PM blakely3dnm notifications@github.com wrote:

I am not familiar with any ICP odometry methods that downsample the clouds to 10 points. Maybe you should write an algorithm that does this, and I will be able to do SLAM with my raspberry pi.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/6869#issuecomment-662182501, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFVCSLB5SRMCYSPGH6IIW3R4Y3FRANCNFSM4PCSNX4A .

[blakely3dnm]

I invite you to research the best ICP odometry algorithms available. I am happy with libpointmatcher, but it uses a few hundred points and 40 iterations under default settings to get good results, and even then those results rely heavily on the environment. Maybe your requirements are not as exact as what I expect from pose estimation, in which case you might just try pose estimation from the IMU. Somewhere on github I saw mention of a person writing a script to get yaw from roll and pitch.

[andrewhodel]

I have no idea why it would be called anything other than measuring and remembering distinct areas, then nobody knows how it works.

Getting position or pose from the IMU only is an exercise in futility.

Andrew

On Tue, Jul 21, 2020 at 7:11 PM blakely3dnm notifications@github.com wrote:

I invite you to research the best ICP odometry algorithms available. I am happy with libpointmatcher, but it uses a few hundred points and 40 iterations under default settings to get good results, and even then those results rely heavily on the environment. Maybe your requirements are not as exact as what I expect from pose estimation, in which case you might just try pose estimation from the IMU. Somewhere on github I saw mention of a person writing a script to get yaw from roll and pitch.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/6869#issuecomment-662184619, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFVCSJYKPO3QP34F6VVOR3R4Y4CHANCNFSM4PCSNX4A .

[andrewhodel]

Surely you have done this with a drone and one of those 10x10px distance sensors.

This is just adding a dimension to that, as they are really only concerned with ground avoidance.

Andrew

On Tue, Jul 21, 2020 at 7:14 PM Andrew Hodel andrewhodel@gmail.com wrote:

I have no idea why it would be called anything other than measuring and remembering distinct areas, then nobody knows how it works.

Getting position or pose from the IMU only is an exercise in futility.

Andrew

On Tue, Jul 21, 2020 at 7:11 PM blakely3dnm notifications@github.com wrote:

I invite you to research the best ICP odometry algorithms available. I am happy with libpointmatcher, but it uses a few hundred points and 40 iterations under default settings to get good results, and even then those results rely heavily on the environment. Maybe your requirements are not as exact as what I expect from pose estimation, in which case you might just try pose estimation from the IMU. Somewhere on github I saw mention of a person writing a script to get yaw from roll and pitch.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/6869#issuecomment-662184619, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFVCSJYKPO3QP34F6VVOR3R4Y4CHANCNFSM4PCSNX4A .

[dorodnic]

@Richard-Elliott - to answer your question, yes, the IR-cut filter of the T265 blocks IR from the L515, so you can use them together.

[Richard-Elliott]

@dorodnic - Thank you @andrewhodel - Sorry if my question did not relate to your original posting, I did not intend to hijack your thread.

[andrewhodel]

@dorodnic - Thank you @andrewhodel - Sorry if my question did not relate to your original posting, I did not intend to hijack your thread.

@Richard-Elliott thank you, sorry to be rude I am just very frustrated with the number of people that keep applying things at such a great extreme to what's really going on and I should just say, off-topic.

[andrewhodel]

@blakely3dnm

Like this, but faster.

[blakely3dnm]

just 10 points will not be nearly enough as you cannot ensure that the same ten points in the scene are sampled unless you already know how the pose has changed (in which case why are you trying to calculate pose de novo?). Picking a random ten points and aligning them to the previous pose will get you a "solution" but it will almost certainly not be the correct one. Effectively you will have aliasing due to the down-sampling and you will align based on that aliasing. Further, even with 1024x768 points using the L515 there are many real world cases where geometric complexity is not sufficient to get a good ICP solution. For instance if a lidar was rotated in front of your curved surface above, along the Y-axis as oriented in your first picture, the scans would all appear identical and you could not resolve any motion.

TL;DR - robust lidar odometry (ICP) is computationally expensive compared to visual odometry. There is simply not enough compute power on a 3 watt device to accomplish this in real time. ICP performs better on a CPU than a VPU/GPU, so even with the hardware in the T265 it would not be possible to do at a reasonable frequency

[andrewhodel]

It has a camera too dude,

http://www.robotics.stanford.edu/~ang/papers/icra10-MultiCameraObjectDetection.pdf

Consider that, with the same camera and different times.

Also, consider that is entirely possible on a .6w raspberry pi zero.

Andrew

On Wed, Jul 22, 2020 at 1:47 PM blakely3dnm notifications@github.com wrote:

just 10 points will not be nearly enough as you cannot ensure that the same ten points in the scene are sampled unless you already know how the pose has changed (in which case why are you trying to calculate pose de novo?). Picking a random ten points and aligning them to the previous pose will get you a "solution" but it will almost certainly not be the correct one. Effectively you will have aliasing due to the down-sampling and you will align based on that aliasing. Further, even with 1024x768 points using the L515 there are many real world cases where geometric complexity is not sufficient to get a good ICP solution. For instance if a lidar was rotated in front of your curved surface above, along the Y-axis as oriented in your first picture, the scans would all appear identical and you could not resolve any motion.

TL;DR - robust lidar odometry (ICP) is computationally expensive compared to visual odometry. There is simply not enough compute power on a 3 watt device to accomplish this in real time. ICP performs better on a CPU than a VPU/GPU, so even with the hardware in the T265 it would not be possible to do at a reasonable frequency

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/6869#issuecomment-662659512, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFVCSOEFLQ7I52HYV273EDR44637ANCNFSM4PCSNX4A .

[andrewhodel]

@blakely3dnm beyond that, I don't care if it arrives as a stream from the camera or is calculated within the API via another call like get_determined_pose_data().

See, I even edited the title to this issue.

[blakely3dnm]

Yea it has a camera with a rolling shutter, a narrow FOV, and no movidius module. If it did have the VPU, it would not perform as well as the T265 for visual odometry, given the camera.

I'm not sure how your tensorflow-like object detection is analogous to the problem here, but in any case, imagine taking a 1024x768 image and downsampling it to a 5x2 image and trying to perform object detection on that. Not gonna work. Data reduction can only go so far.

I get good ICP odometry with the L515 in 640x480 mode and taking a random subset of half the points. With those settings on a Ryzen 9 CPU I get 15 Hz pose information using L515 data, but I still have to be careful about how I move the device to avoid getting lost.

This is just my take after actually having tried home made versions of VIO and ICP. If you think it should work a different way in theory, fine, give it a try, and I will be watching your github for some revolutionary SLAM algorithms.

[andrewhodel]

If pulling rope is revolutionary then I don't know what to say.

All you are doing is finding the same point that it was attached to before and expecting there to be a clear path somewhere on the image.

Not to mention the IMU data.

On Wed, Jul 22, 2020 at 2:04 PM blakely3dnm notifications@github.com wrote:

Yea it has a camera with a rolling shutter, a narrow FOV, and no movidius module. If it did have the VPU, it would not perform as well as the T265 for visual odometry, given the camera.

I'm not sure how your tensorflow-like object detection is analogous to the problem here, but in any case, imagine taking a 1024x768 image and downsampling it to a 5x2 image and trying to perform object detection on that. Not gonna work. Data reduction can only go so far.

I get good ICP odometry with the L515 in 640x480 mode and taking a random subset of half the points. With those settings on a Ryzen 9 CPU I get 15 Hz pose information using L515 data, but I still have to be careful about how I move the device to avoid getting lost.

This is just my take after actually having tried home made versions of VIO and ICP. If you think it should work a different way in theory, fine, give it a try, and I will be watching your github for some revolutionary SLAM algorithms.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/6869#issuecomment-662667986, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFVCSLSNOWNS2NRJQSBZU3R45A5PANCNFSM4PCSNX4A .

[blakely3dnm]

Again, you can only find the same point as before if you already know the pose (as you can't "pull" on lidar measurements) . So you cant use that method to find the pose.

[andrewhodel]

Regardless of any "novel" idea, your "robust ICP algorithm" should already be implemented as an api call within librealsense as the camera itself is already sending the accel/gyro data, lidar stream, infrared stream and video stream.

Why has that not been done?

Andrew

On Wed, Jul 22, 2020 at 2:07 PM Andrew Hodel andrewhodel@gmail.com wrote:

If pulling rope is revolutionary then I don't know what to say.

All you are doing is finding the same point that it was attached to before and expecting there to be a clear path somewhere on the image.

Not to mention the IMU data.

On Wed, Jul 22, 2020 at 2:04 PM blakely3dnm notifications@github.com wrote:

Yea it has a camera with a rolling shutter, a narrow FOV, and no movidius module. If it did have the VPU, it would not perform as well as the T265 for visual odometry, given the camera.

I'm not sure how your tensorflow-like object detection is analogous to the problem here, but in any case, imagine taking a 1024x768 image and downsampling it to a 5x2 image and trying to perform object detection on that. Not gonna work. Data reduction can only go so far.

I get good ICP odometry with the L515 in 640x480 mode and taking a random subset of half the points. With those settings on a Ryzen 9 CPU I get 15 Hz pose information using L515 data, but I still have to be careful about how I move the device to avoid getting lost.

This is just my take after actually having tried home made versions of VIO and ICP. If you think it should work a different way in theory, fine, give it a try, and I will be watching your github for some revolutionary SLAM algorithms.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/6869#issuecomment-662667986, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFVCSLSNOWNS2NRJQSBZU3R45A5PANCNFSM4PCSNX4A .

[andrewhodel]

That is absolutely untrue.

It is about the refresh rate and the speed of movement of the device.

The same is true of the T265, it's not going to work at light speed and it isn't because of the IMU data.

On Wed, Jul 22, 2020 at 2:10 PM blakely3dnm notifications@github.com wrote:

Again, you can only find the same point as before if you already know the pose (as you can't "pull" on lidar measurements) . So you cant use that method to find the pose.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/6869#issuecomment-662670669, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFVCSM5STFG5ILQAW2M3NDR45BSBANCNFSM4PCSNX4A .

[andrewhodel]

Look at canvas.io

How do you think the iPad pro does it, magic?

On Wed, Jul 22, 2020 at 2:11 PM Andrew Hodel andrewhodel@gmail.com wrote:

That is absolutely untrue.

It is about the refresh rate and the speed of movement of the device.

The same is true of the T265, it's not going to work at light speed and it isn't because of the IMU data.

On Wed, Jul 22, 2020 at 2:10 PM blakely3dnm notifications@github.com wrote:

Again, you can only find the same point as before if you already know the pose (as you can't "pull" on lidar measurements) . So you cant use that method to find the pose.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/6869#issuecomment-662670669, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFVCSM5STFG5ILQAW2M3NDR45BSBANCNFSM4PCSNX4A .

[andrewhodel]

The lidar in the iPad is honestly near having 10px of density, LOL.

On Wed, Jul 22, 2020 at 2:14 PM Andrew Hodel andrewhodel@gmail.com wrote:

Look at canvas.io

How do you think it iPad pro does it, magic?

On Wed, Jul 22, 2020 at 2:11 PM Andrew Hodel andrewhodel@gmail.com wrote:

That is absolutely untrue.

It is about the refresh rate and the speed of movement of the device.

The same is true of the T265, it's not going to work at light speed and it isn't because of the IMU data.

On Wed, Jul 22, 2020 at 2:10 PM blakely3dnm notifications@github.com wrote:

Again, you can only find the same point as before if you already know the pose (as you can't "pull" on lidar measurements) . So you cant use that method to find the pose.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/6869#issuecomment-662670669, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFVCSM5STFG5ILQAW2M3NDR45BSBANCNFSM4PCSNX4A .

[blakely3dnm]

It is true. knowing the location of arbirtary points in scene is the same as knowing the pose.

im guessing canvas is very similar in principle to the tracking in the T265. It probably makes a registered depth image and only uses few points from the scene. You could argue that the L515 should have had a movidius VPU so it could find pose using the RGBD stream, but then you wouldn't need to buy a T265 ;)

[andrewhodel]

The trouble is the t265 is no good at night.

There's no reason this shouldn't be working.

On Wed, Jul 22, 2020 at 2:21 PM blakely3dnm notifications@github.com wrote:

It is true. knowing the location of arbirtary points in scene is the same as knowing the pose.

im guessing canvas is very similar in principle to the tracking in the T265. It probably makes a registered depth image and only uses few points from the scene. You could argue that the L515 should have had a movidius VPU so it could find pose using the RGBD stream, but then you wouldn't need to buy a T265 ;)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/6869#issuecomment-662675777, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFVCSK2KP4A5UWJEH4LZHDR45C3ZANCNFSM4PCSNX4A .

[blakely3dnm]

canvas is no good at night either, nor would be the L515 RGBD stream. ICP alone, as far as I know, is not as fast as calculating pose from RGBD. Feature extraction from laser scans is a developing field, so that could change.

But I'm just a hobbyist, what do I know. I will stop spamming the repo now.

[andrewhodel]

At the refresh rate of the l515 it should be much faster at calculating pose that a video stream.

I just think nobody has thought about it.

On Wed, Jul 22, 2020 at 2:28 PM blakely3dnm notifications@github.com wrote:

canvas is no good at night either, nor would be the L515 RGBD stream. ICP alone, as far as I know, not as fast as calculating pose from RGBD. Feature extraction from laser scans is a developing field, so that could change.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/6869#issuecomment-662679159, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFVCSOC5S7SYP6IJRJXXMDR45DWTANCNFSM4PCSNX4A .

[andrewhodel]

I cannot believe Intel wouldn’t put pose data on a lidar camera!

On Wed, Aug 5, 2020 at 9:00 AM Sergey Dorodnicov notifications@github.com wrote:

Closed #6869 https://github.com/IntelRealSense/librealsense/issues/6869.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/6869#event-3625097450, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFVCSL556PMCRUK3RZMWLDR7FXYPANCNFSM4PCSNX4A .

[malleshamdasari]

Yea it has a camera with a rolling shutter, a narrow FOV, and no movidius module. If it did have the VPU, it would not perform as well as the T265 for visual odometry, given the camera.

I'm not sure how your tensorflow-like object detection is analogous to the problem here, but in any case, imagine taking a 1024x768 image and downsampling it to a 5x2 image and trying to perform object detection on that. Not gonna work. Data reduction can only go so far.

I get good ICP odometry with the L515 in 640x480 mode and taking a random subset of half the points. With those settings on a Ryzen 9 CPU I get 15 Hz pose information using L515 data, but I still have to be careful about how I move the device to avoid getting lost.

This is just my take after actually having tried home made versions of VIO and ICP. If you think it should work a different way in theory, fine, give it a try, and I will be watching your github for some revolutionary SLAM algorithms.

@blakely3dnm, can you please hint how did you get the ICP odometry with L515? I mean what do you get from L515 and is there an opensources ICP already that I can just input the data from L515?

[blakely3dnm]

@blakely3dnm, can you please hint how did you get the ICP odometry with L515? I mean what do you get from L515 and is there an opensources ICP already that I can just input the data from L515?

you can use libpointmatcher to calculate the transform between two laser scans and use that transform as the odometry. If you are already using ROS you can use rtabmap-ros compiled with libpointmatcher to do this. I also use the IMU as a constraint on motion, but the point cloud from the L515 is all you need.

[malleshamdasari]

Sorry, I am a bit new to this. I am recording the pcd using L515 now. Are you saying that rtabmap-ros with libpointmatcher can take this pcd as input and it can give me the pose directly?

[blakely3dnm]

If you do not already use ROS, it may not be the simplest solution. That being said, with the correct settings realsense-ros can publish the lidar data from the L515 as a pointcloud at 30 hz . The rtabmap-ros odometry node can then subscribe to those lidar frames and find the transform between each frame by icp. The odometry node in turn publishes an odometry topic which contains the pose information (roll, pitch, yaw, translation), though it may or may not be presented in that format.

[malleshamdasari]

Thanks @blakely3dnm. So, I tried the following commands to see their odometry.

roslaunch realsense2_camera opensource_tracking.launch enable_infra:=true filters:=pointcloud
rostopic echo /rtabmap/odom

And I ran your launch too that you provided here.

I see the 6DoF translation and orientation in the output using both launches, but it is extremely fragile for the movement it seems. It fails and gives all zero's position if move the Lidar sensor by small motion. Do you know if this is a problem/or the limitation of rtabmap odometry or realsense-ros?

[blakely3dnm]

That launch file was my first attempt to make visual odometry work, but there are several parameters that still need to be optimized. I am able to map my office with those settings and continual smooth motion but the performance will depend on the visual features in the scene. A good starting point to optimize those settings is here: http://wiki.ros.org/rtabmap_ros/Tutorials/Advanced%20Parameter%20Tuning But instead of doing that I would recommend ICP using this launch file. With those settings I get pose estimation at ~30 hz with a Ryzen 4900HS CPU. There are a few cases where ICP will get lost, such as flat walls with no landmarks on them. Keeping geometric complexity in the sensors field of view is key.

[andrewhodel]

So blakely, is the ros libpointmatcher the never before built revolutionary method that has never been attempted for pose from lidar frames you were talking about earlier in the thread?

On Wed, Sep 23, 2020 at 11:44 PM blakely3dnm notifications@github.com wrote:

That launch file was my first attempt to make visual odometry work, but there are several parameters that still need to be optimized. I am able to map my office with those settings and continual smooth motion but the performance will depend on the types of visual features in the scene. A good starting point to optimize those settings is here: http://wiki.ros.org/rtabmap_ros/Tutorials/Advanced%20Parameter%20Tuning

But instead of doing that I would recommend using ICP using this launch file https://github.com/introlab/rtabmap/issues/574#issuecomment-689149729. With those settings I get pose estimation at ~30 hz with a Ryzen 4900HS CPU. There are a few cases where ICP will get lost, such as flat walls with no landmarks on them. Keeping geometric complexity in the sensors field of view is key.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/6869#issuecomment-698125872, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFVCSP2MJUU4OU5Q36KP3LSHLMEDANCNFSM4PCSNX4A .

[malleshamdasari]

That launch file was my first attempt to make visual odometry work, but there are several parameters that still need to be optimized. I am able to map my office with those settings and continual smooth motion but the performance will depend on the visual features in the scene. A good starting point to optimize those settings is here: http://wiki.ros.org/rtabmap_ros/Tutorials/Advanced%20Parameter%20Tuning But instead of doing that I would recommend ICP using this launch file. With those settings I get pose estimation at ~30 hz with a Ryzen 4900HS CPU. There are a few cases where ICP will get lost, such as flat walls with no landmarks on them. Keeping geometric complexity in the sensors field of view is key.

Thank you Blakely. I will try out with the Tutorial and check out the ICP launch that you provided. I will continue on the other thread.