Detecting sidewalks

[sivashankar28]

I am trying to detect the sidewalk plane and use it to detect potholes and curbs so that I can view it on a screen when travelling on my wheelchair. Does your "Depth Mask" code help me execute this? I am trying to ensure it detects the ground plane first by using your "tracking.py" file but unsure of how to manipulate this further to detect curbs or potholes.

Any help would be very much appreciated! Thank you!

[JeremyBYU]

Hey thanks for checking out the repo. I hope Polylidar can be useful for and your situation and will do my best to help. A couple notes.

• Tracking.py is meat to be used with a d435 and t265. The t265 is meant to just give orientation (gravity vector) so that the point cloud can be aligned to the [0,0,1] vector. This is necessary in this current version of Polylidar to extract polygons. The capture.py file instead uses plane fitting. Other than that they should be the same.
• This RealSense camera gives really noisy pointclouds. You have to use a lot of spatial and temporal filtering. If you are moving in your wheel chair this may be an issue. As in the temporal filter cant be too aggressive.
• I am doing a major update to Polylidar now, called Polylidar3D. This update will do a much better job of identifying the floor normal and smoothing the point cloud. It can also capture all the dominant planes in a scene. However it’s just not ready yet.

As for pot holes they can be found but they would need to be pretty deep. Small shallow holes will just not be captured because of the noise in the sensor. I would think the pothole would need to be several inches deep (3-5?).

[sivashankar28]

Thank you for the prompt reply Jeremy! I am currently using the D415 camera and the T265 camera. I am have had a lot of issues with trying to get around dealing with sunlight as the exposure gets too high and the image saturates. I have tried to research different methods of applying optical filters but I have had no success. I look forward to the Polylidar3D as your work is fantastic! Even potholes that are 1 inch deep are an issue for wheelchair users so I would love to be able to detect it. As this would be a shallow pothole, would you recommend another sensor that would work better for outdoor use to achieve my goal? (maybe the Zed2 camera)

[JeremyBYU]

I'm sorry about the struggles with the sensor, I know exactly what you are talking about! As for the sunlight - One thing that I have also done is drastically reduce the exposure time. I believe that I have even dropped it to 2 milliseconds once outside. You probably have already done that but thought I would bring it up just in case. Optical filters would be even better I am not sure what or where to buy them. If you ever figure that out let me know!

As for a different sensor. I have just purchased the L515 from Intel. It is a solid state lidar device and seems to have much less noise. Around 5mm of noise at 1m and 14mm of noise at 9 meters! At that lower noise level I think 1-inch pot holes could be detected. My sensor should arrive in July (fingers crossed). I will let you know how effective it is and will even do some pothole experiments.

[sivashankar28]

I was debating about using the L515 or the Zed 2 camera but was unsure which was better? Have you done any cross-comparisons with other devices? If I have any success with filters, I shall let you know! For now, I wanted to use the same code but to detect a vertical plane instead of the ground plane and eventually merge the codes together. Are you able to advise me on how to change your current code?

[JeremyBYU]

As for sensors I have tested this extensively with velodyne lidar and also airborne lidar point clouds. Worked really good on them, they have very little noise compared to vision depth sensors,

As it currently stands the tracking.py hardcodes to extract the floor. So I think you would want to use capture.py and then modify this function: calculate_plane_normal(ground_patch)

You would have to send it a patch (basically a planar collection of points) of the surface you are interested in. That would be up to you how to get that. It will return the normal of that plane and then rotate the point cloud to align with this normal. However there are also several caveats where it might not work perfectly.

The frustrating thing is that my new update makes this all work fine and not having as many issues. There is no hardcoded anything and is meant to extract all the surfaces. But not quite ready yet but should be sometime this month.

[sivashankar28]

Thank you for the advice! I shall work on it and see where I get with it. Looking forward to seeing your work when you finish the polylidar 3D