project: sidewalk navigation

[Ezward]

The idea here is to create an autopilot that can flexibly drive on a network of sidewalks, making decisions at each intersection. The goal would be an autopilot that could be reprogrammed to drive a specific path or circuit within a sidewalk network simply by giving it a list of turns.

Create an autopilot that can

• drive on a sidewalk
• recognize an intersection
• make a decision at the intersection on which way to proceed
• make a turn or drive through an intersection based on a turn decision

The first version of the autopilot would do this without GPS. So if a outdoor sidewalk network is not available, but an indoor network of hallways is available, then the same kind of autopilot could be created for the hallway navigation.

The 'make a decision' piece would be configurable

• if a set of turns are recorded (just a text file with lines that give the verb; 'right', 'left', 'though'), then the autopilot would simply use the next turn command and execute it. In this way a path or circuit within the sidewalk network could be constructed by editing a turn file.
• the default case would stop and wait for user input on the direction. In that case, if turn recording was turned on, then the users choice would be recorded into the turn file.

Elaborations (these can all be done as separate projects)

• Create an autopilot that would recognize a sign at an intersection to decide how to turn.
• Create an autopilot that will slow/stop if it detects a person on the path in front of it and proceed when the path is cleared.
• Integrate an inexpensive GPS such that it can be use to create a map of the network. We are specifically NOT using high resolution RTK GPS;
  • we want to this be accessible to the most number of people
  • we don't want GPS to be the primary navigation sensor, rather it would be used to map/detect a place of interest).
  • this is much more like how GPS is used in real car navigation/autonomy.
• Create an autopilot that can autonomously map a sidewalk network (assuming a cheap GPS for location) within a given Geo-boundary. So this is essentially a maze walker; the car can find the complete sidewalk network on it's own.
• Create an autopilot that, given a map of the sidewalk network and a list of 'addresses' and their locations on the map, can plan a delivery; drive to an address for a 'pickup' (just wait for a user input as the pickup), plan a path to the destination, drive to the destination and do a drop off (wait for user input).

[TCIII]

@Ezward,

Might this be accomplished by the use of an inexpensive Lidar Scanner, such as the Slamtech A1-A3 family of Lidar scanners, along with a non-RTK GPS as the accuracy of non-RTK GPSs is at best 2.5 meters and GPS positional knowledge (drift) can deteriorate over time of use?

Regards, TCIII

[Ezward]

@Ezward,

Might this be accomplished by the use of an inexpensive Lidar Scanner, such as the Slamtech A1-A3 family of Lidar scanners, along with a non-RTK GPS as the accuracy of non-RTK GPSs is at best 2.5 meters and GPS positional knowledge (drift) can deteriorate over time of use?

Regards, TCIII

A Lidar would help with the slow/stopping if a person or other obstacle was detected. I think it would not help with mapping the sidewalk outside (assuming there are no permanent obstacles on the sidewalk as the would void the 'walk' promise). But inside a network of hallways it could be very useful because 1) it could be used to know the precise width as the car drives 2) assuming the walls are perpendicular to each other you could also probably use it to figure out the car's heading relative to the walls 3) hallways probably have more semi-permanent obstacles I think, so they could be mapped.

[Ezward]

What if we could correct the gps position based on recognizing a milestone on the path that has been well localized?

We know from using non-RTK GPS with the path follow template that non-RTK can be useful if it can be 'zeroed' out to subtract gps drift at the start of a race; then the drift during a race may be small enough to not be a problem. So if we can correct the gps position periodically as we drive, then even a cheap GPS could be used to navigate the path.

Now thinking about this in the sidewalk navigation scenario, what if you could correct your position when you come to a known milestone on the course. For instance what if each intersection was well localized ahead of time; perhaps by using satellite data/images or simply by using an RTK gps to measure each intersection's position. Of if not an intersection then a sign along the way; the sign itself might indicate it's position using a QR-Code; the vehicle, if it recognizes the QR code, will estimate it's position relative to the sign and then use that and the value contained in the QR code to reset it's gps position. That could be fun.

[TCIII]

@Ezward,

An excellent suggestion.

This project is beginning to sound like the Hobby version of the Lunar Rover Challenge.

TCIII

[Ezward]

Another simpler project might be to create a turn-by-turn navigation; like google navigate. A gps map of the sidewalk network would be given.

1. Given a set of turns (in the turn file) provide the human driving with turn by turn instructions; this could be done with LEDs for left and right turns and maybe a green led for straight. The first version would stop the car if the user makes a wrong turn. The instructions will indicate when the car has reached the destination.
2. Add a path finding algorithm; given the map and a list of points of interest within the sidewalk network; when given a destination point of interest calculate an optimal path to the point of interest and start turn by turn navigation.

[TCIII]

@Ezward,

As an alternative to using GPS for location, we could use extremely high resolution wheel encoders.

I have a 6WD robot chassis where the middle set of wheels is driven by motors with a high resolution wheel encoder on the back of the drive motor. Due to the high resolution of the motor encoders, I can achieve an output of 0.3142 mm/ encoder tick of travel. Additionally, since the encoders are on motors in the middle of the chassis, left and right turns as well as pivot turns are accurately recorded. The Rpi 4B mounted on this 6WD chassis is running the 921-path-follow-in-simulator branch and does so quite well given the high resolution knowledge of wheel travel.

Without the use of GPS-RTK, the accuracy of a Ublox NEO-M9N GPS, without dead reckoning, is at best 2.5 meters and can drift with time.

Comments?

[Ezward]

• Here is an article on segmentation models to recognize sidewalks from a small car, like a donkeycar; Robot following a walkway using image segmentation That article indicates the raspberry pi is too slow to run the ENet Segmentation model using the full resolution images; it only runs 1 frame each 6 seconds. Solution would be to retrain on smaller images and/or run the inference on a neural processor of some kind.
• Article showing image segmentation running on Jetson Nano at 45fps on 320x320 images Realtime semantic segmentation jetson nano python, related repo jetson inference
• Autonomous Off-Road Vehicle Using Image Segmentation & Nvidia Jetson Nano
• Lane Line Detection Algorithm Based on Instance Segmentation "The experimental results show that the Acc value of the lane line detection algorithm based on instance segmentation is 96.7%, and the FPS is 77.5 fps/s. The detection speed deployed on the embedded platform Jetson Nano reaches 27 fps/s."

[Ezward]

@Ezward,

As an alternative to using GPS for location, we could use extremely high resolution wheel encoders.

I have a 6WD robot chassis where the middle set of wheels is driven by motors with a high resolution wheel encoder on the back of the drive motor. Due to the high resolution of the motor encoders, I can achieve an output of 0.3142 mm/ encoder tick of travel. Additionally, since the encoders are on motors in the middle of the chassis, left and right turns as well as pivot turns are accurately recorded. The Rpi 4B mounted on this 6WD chassis is running the 921-path-follow-in-simulator branch and does so quite well given the high resolution knowledge of wheel travel.

Without the use of GPS-RTK, the accuracy of a Ublox NEO-M9N GPS, without dead reckoning, is at best 2.5 meters and can drift with time.

Comments?

@TCIII I think encoders are really great to maintain a path between GPS updates, which might be slow (7 per second) and as a backup for short periods of time if GPS fails. However because they accumulate error we need some external way to localize to effectively reset the accumulated error. So we need GPS or visual odometry or something.

[Ezward]

Potential data sets for a sidewalk segmentation model

• https://www.researchgate.net/publication/362550245_A_Dataset_for_Temporal_Semantic_Segmentation_Dedicated_to_Smart_Mobility_of_Wheelchairs_on_Sidewalks
• https://huggingface.co/datasets/segments/sidewalk-semantic
• https://ras.papercept.net/images/temp/IROS/files/1873.pdf

[Ezward]

Free space segmentation for navigating sidewalks: https://docs.nvidia.com/isaac/archive/2020.1nx/packages/freespace_dnn/doc/freespace_segmentation.html

[Ezward]

• Tensorflow tutorial using U-Net segmentation model; https://www.tensorflow.org/tutorials/images/segmentation
• Tensorflow-lite image segmentation https://www.tensorflow.org/lite/examples/segmentation/overview
• Tensorflow-lite U-Net: tensorflow-lite https://github.com/PINTO0309/TensorflowLite-UNet
• QEngineering Tensorflow-Lite segmentation on RaspberryPi at 7fps:
  • https://www.youtube.com/watch?v=Kh9DLMgCIIE
  • https://github.com/Qengineering/TensorFlow_Lite_Segmentation_RPi_64-bit

[Ezward]

If we were to do this indoors using corridors, where is a paper on corridor navigation; https://www.semanticscholar.org/paper/Vision-based-assistance-for-wheelchair-navigation-Pasteau-Krupa/3306150d768c833864c0fbd243ab06fd6a6af70e

[Ezward]

Visual odometry and depth camera for wheel chair navigation; https://www.cs.ubc.ca/~mack/Publications/ViswanathanIROSWorkshop2011.pdf

[Ezward]

Good article on image segmentation techniques; https://www.v7labs.com/blog/image-segmentation-guide

[Alex-Beh]

Interesting discussion. May I know how is the plan on going now?

[TCIII]

@Alex-Beh,

Due to all the prior programming commitments, probably on a back burner at the moment.

TCIII