[Feature Request] Temporary Static Obstacles Detection and Map Update Using RL

[BCKSELFDRIVEWORLD]

Feature Request

Feature Description

This feature aims to enhance indoor mobile robots by enabling dynamic detection of unforeseen temporary static obstacles and updating the map using Markov Decision Processes (MDP) with Reinforcement Learning (RL) algorithms. This will allow robots to identify and record these obstacles on the map after repeated encounters, optimizing their maneuvering and reducing task completion time. By increasing the cost values of cells with persistent obstacles, robots can plan more efficient paths and avoid unnecessary maneuvers.

Use Case

Currently, robots create their initial paths based on a global costmap, but they often encounter unforeseen static obstacles that require frequent maneuvering, increasing task duration. This feature will enable robots to dynamically adapt to such obstacles, improving their efficiency and reducing the time needed to complete tasks. This is particularly useful in environments where temporary static obstacles are common, such as warehouses, hospitals, and offices.

Implementation Details

Using Reinforcement Learning and MDP:

Defining the MDP:

• States (S): The robot's position and the state of the surrounding environment (e.g., x, y coordinates, positions of obstacles).
• Actions (A): Possible actions the robot can take (e.g., move forward, backward, left, right, stop).
• Transition Function (T): Probabilities of transitioning from one state to another after taking an action.
• Rewards (R): Rewards or penalties the robot receives for actions (e.g., reward for moving closer to the goal, penalty for encountering an obstacle).

Training the RL Algorithm:

• Q-Learning or Deep Q-Networks (DQN): Algorithms to learn which action to take in different states.
• Reward Structure: The robot receives a negative reward for encountering a temporary obstacle. If the robot encounters the same obstacle a certain number of times (e.g., 3 times), it assumes the obstacle is persistent and updates the map by increasing the cost of that cell.
• Experience Replay: Allows the robot to learn from past experiences to improve the effectiveness of the algorithm.

Map Update:

• Increase the cost values of cells where persistent obstacles are detected.
• Apply these updates to the global costmap.
• Plan new routes based on the updated costmap.

Path Planning and Maneuver Optimization:

• Use the updated costmap in the robot’s path planning algorithm.
• The robot preemptively knows the locations of previously encountered persistent obstacles.
• Optimize the route to minimize unnecessary maneuvers.

Expected Benefits

• Efficiency: Robots will navigate more efficiently by avoiding frequently encountered obstacles.
• Safety: Reduced unnecessary maneuvers will lead to safer navigation.
• Time-saving: Overall task completion time will be reduced, enhancing productivity.

I believe this feature is logical and feasible. I have a few questions in my mind that need clarification, but overall, this concept seems promising to me. If the community is as enthusiastic about this feature as I am, I would love to work on this topic.

[SteveMacenski]

Certainly an interesting idea! Have you worked on something similar before and have an intuition that this would work well? Having something to better address dynamic obstacles at a costmap abstraction level would be fabulous.

I'd love to see a prototype of this in action on your github before we discuss integrating it directly into the Nav2 stack to make sure it performs well enough and sufficiently general for folks' use case. If its not super portable, its still a very useful project to have in the ecosystem for people to use if they can or adapt it!